From strawman at astraw.com Tue Apr 3 05:07:43 2007 From: strawman at astraw.com (Andrew Straw) Date: Mon, 02 Apr 2007 20:07:43 -0700 Subject: [Distutils] [ANN] stdeb 0.2.a1 Message-ID: <4611C4FF.3020001@astraw.com> I would like to announce the release of stdeb 0.2.a1. = What is it? = stdeb http://stdeb.python-hosting.com/ ("setuptools debian") produces Debian source packages from Python packages via a new distutils command, sdist_dsc, which produces a Debian source package of a Python package. Automatic defaults are provided for the Debian package, but many aspects of the resulting package can be customized via a configuration file. = What's new? = 1) python-central support 2) lots of testing resulting in lots of bug fixes = The gory details = Changelog for Release 0.2.a1 (2007-04-02) 2007-04-02 Make default distribution 'unstable' 2007-03-28 Removed versioned Python interpreter from packages in /usr/bin and /usr/lib. Bump version to 0.2.a1. Include version in autogenerated debian/rules and debian/changelog. 2007-03-15 pycentral support enabled by default 2007-02-04 fix for cases in which setup.py wants __file__ variable 2006-09-07 added --patch-posix to support posix-mode patches 2006-09-07 always extract source using "dpkg-source -x" 2006-09-07 moved working directory for patches one level shallower (may require patch level 1 instead of patch level 0) 2006-08-25 added Forced-Upstream-Version field in .cfg file 2006-08-22 patching source works when --premade-distfile option is given but distfile must be regenerated 2006-08-22 patch level supported via --patch-level option 2006-08-22 Add support for bzip2 tarballs 2006-08-13 Generate source by copying tree, not "python setup.py sdist" 2006-08-13 Patches can be applied prior to first call to setup.py 2006-07-21 Allow patches to upstream source via Stdeb-Patch-File in .cfg file. 2006-07-21 Don't bother trying to rename .egg-info directories. 2006-07-09 Ability to specify MIME-Desktop-Files, MIME-File, and Shared-MIME-File in configuration file. 2006-07-09 Use of dh_python for generation of postinst and prerm scripts. 2006-07-05 Auto-generation of debian/postinst and debian/prerm scripts that create and remove .pyc and .pyo files. 2006-06-21 Upstream .tar.gz files can be used exactly, allowing md5sums to match. 2006-06-21 Expanded source directory is only deleted when commandline option is set. From janwijbrand at gmail.com Wed Apr 4 17:20:47 2007 From: janwijbrand at gmail.com (Jan-Wijbrand Kolman) Date: Wed, 4 Apr 2007 17:20:47 +0200 Subject: [Distutils] HTTP authentication support In-Reply-To: <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> Message-ID: > The patch is minor enough that I added it to 0.6; it only affects URLs that > would have caused today's easy_install to crash with an httplib.InvalidURL > error, anyway. > > The easiest way to use it is to use a -f or dependency_links URL that > points to a page with links to the actual downloads. As long as they are > the same scheme and host, they'll inherit the credentials from the original > URL. Trying this now, using setuptools-0.6c6dev_r54241, but I must be doing something wrong here, because I cannot get it to work. What I do is: easy_install -f \ https://user:secret at myhost.com/trac/wiki/PackageIndex SomePackage This index page requires authentication and easy_install can indeed read the index page with the supplied credentials. It finds the correct download URL too, i.e.: https://myhost.com/svn/somepackage#egg=SomePackage It was my understanding that easy_install would reuse the credentials as specified for the index page URL, since both host and scheme are identical for the download and index page URLs. However downloading fails, with a 401 Authorization Required error. As I said, it could very well be I'm doing something wrong here. Any hint is highly appreciated. kind regards, jw -- Jan-Wijbrand Kolman From pje at telecommunity.com Wed Apr 4 17:40:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 04 Apr 2007 11:40:59 -0400 Subject: [Distutils] HTTP authentication support In-Reply-To: References: <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070404114004.02e22708@sparrow.telecommunity.com> At 05:20 PM 4/4/2007 +0200, Jan-Wijbrand Kolman wrote: >What I do is: > > easy_install -f \ > https://user:secret at myhost.com/trac/wiki/PackageIndex SomePackage > >This index page requires authentication and easy_install can indeed >read the index page with the supplied credentials. It finds the >correct download URL too, i.e.: > > https://myhost.com/svn/somepackage#egg=SomePackage > >It was my understanding that easy_install would reuse the credentials >as specified for the index page URL, since both host and scheme are >identical for the download and index page URLs. However downloading >fails, with a 401 Authorization Required error. > >As I said, it could very well be I'm doing something wrong here. Any >hint is highly appreciated. Is the download link a *relative* URL, or absolute? If it's absolute, the credentials won't be there. From janwijbrand at gmail.com Thu Apr 5 09:38:19 2007 From: janwijbrand at gmail.com (Jan-Wijbrand Kolman) Date: Thu, 5 Apr 2007 09:38:19 +0200 Subject: [Distutils] HTTP authentication support In-Reply-To: <5.1.1.6.0.20070404114004.02e22708@sparrow.telecommunity.com> References: <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> <5.1.1.6.0.20070404114004.02e22708@sparrow.telecommunity.com> Message-ID: > Is the download link a *relative* URL, or absolute? If it's absolute, the > credentials won't be there. They were absolute URLs. I just tried with relative URLs and they *do* indeed inherit the credentials supplied for the package index URL. So, thanks for the hint! A remark, from, say, a usability standpoint: The fact that you need to have relative URLs in order to inherit the credentials was not clear to me from your pervious post to this thread. There you explain as long as host and scheme are identical, the credentials should work. And well, for relative URLs this is automatically true of course :-) End users like me, would actually expect the described behaviour as this is how e.g. web browsers handle these types of authentication. So, this needs either a very clear explanation somewhere, or maybe a little bit of work on easy_install, in order to "cache" the credentials on a host/scheme basis. Anyway, thanks again! kind regards, jw -- Jan-Wijbrand Kolman From pje at telecommunity.com Thu Apr 5 19:16:35 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Apr 2007 13:16:35 -0400 Subject: [Distutils] HTTP authentication support In-Reply-To: References: <5.1.1.6.0.20070404114004.02e22708@sparrow.telecommunity.com> <5.1.1.6.0.20061228004031.026b3788@sparrow.telecommunity.com> <5.1.1.6.0.20061228012226.02830338@sparrow.telecommunity.com> <87lkksjyqa.fsf@exponent.kumachan.no-ip.org> <5.1.1.6.0.20061230103134.02a8b8b8@sparrow.telecommunity.com> <5.1.1.6.0.20070404114004.02e22708@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070405131441.029926f8@sparrow.telecommunity.com> At 09:38 AM 4/5/2007 +0200, Jan-Wijbrand Kolman wrote: >So, this needs either a very clear explanation somewhere, or maybe a >little bit of work on easy_install, in order to "cache" the >credentials on a host/scheme basis. I've added an entry to the "tips" section of the documentation (in my local copy, not checked in anywhere yet): Password-Protected Sites ------------------------ If a site you want to download from is password-protected using HTTP "Basic" authentication, you can specify your credentials in the URL, like so:: http://some_userid:some_password at some.example.com/some_path/ You can do this with both index page URLs and direct download URLs. As long as any HTML pages read by easy_install use *relative* links to point to the downloads, the same user ID and password will be used to do the downloading. From venkatbo at yahoo.com Thu Apr 5 20:00:21 2007 From: venkatbo at yahoo.com (Venkat Bommakanti) Date: Thu, 5 Apr 2007 11:00:21 -0700 (PDT) Subject: [Distutils] How to prevent some .egg-info files being "touched" during a build/install process In-Reply-To: Message-ID: <625315.78934.qm@web30506.mail.mud.yahoo.com> Hi folks, I have an app that I'm using setuptools to build/install. During the build process (I think), it "touches" the following files: ... myapp/src/app/app.egg-info/PKG-INFO myapp/src/app/app.egg-info/SOURCES.txt myapp/src/app/app.egg-info/dependency_links.txt myapp/src/app/app.egg-info/requires.txt myapp/src/app/app.egg-info/top_level.txt ... even though none of the files in the app were modified. As a result the source control system thinks there are new versions of these files. THough they are not marked changed by the source control system, the fact that they are being flagged as being "tocuhed" is causing confusion amongst folks in my team unrelated to my comp - when they see that their changes elsewhere - is affecting my comp. Since these files are deployed as part of the install step, looks like they have to be included under source control. Is there a way to ask setuptools to not "touch" these files if nothing underlying has changed. I can understand SOURCES.txt would have to change when a new source file is added or an old one removed, but others in the above list pretty much remain the same within a given release. Thanks for the help, /venkat ____________________________________________________________________________________ Food fight? Enjoy some healthy debate in the Yahoo! Answers Food & Drink Q&A. http://answers.yahoo.com/dir/?link=list&sid=396545367 From matt at matt-good.net Thu Apr 5 21:05:25 2007 From: matt at matt-good.net (Matt Good) Date: Thu, 05 Apr 2007 15:05:25 -0400 Subject: [Distutils] How to prevent some .egg-info files being "touched" during a build/install process In-Reply-To: <625315.78934.qm@web30506.mail.mud.yahoo.com> References: <625315.78934.qm@web30506.mail.mud.yahoo.com> Message-ID: <1175799925.5982.139.camel@nny> On Thu, 2007-04-05 at 11:00 -0700, Venkat Bommakanti wrote: > Hi folks, > > I have an app that I'm using setuptools to > build/install. During the build process (I think), it > "touches" the following files: > ... > myapp/src/app/app.egg-info/PKG-INFO > myapp/src/app/app.egg-info/SOURCES.txt > myapp/src/app/app.egg-info/dependency_links.txt > myapp/src/app/app.egg-info/requires.txt > myapp/src/app/app.egg-info/top_level.txt > ... > even though none of the files in the app were > modified. > As a result the source control system thinks there are > new versions of these files. THough they are not > marked changed by the source control system, the fact > that they are being flagged as being "tocuhed" is > causing confusion amongst folks in my team unrelated > to my comp - when they see that their changes > elsewhere - is affecting my comp. > > Since these files are deployed as part of the install > step, looks like they have to be included under source > control. Is there a way to ask setuptools to not > "touch" these files if nothing underlying has changed. I typically add the .egg-info folder to the "ignored" files in version control since the files in there will be automatically regenerated by the build process. Unless you have custom files that you put in the .egg-info folder (which I think is uncommon) you can probably just leave that folder out of version control too. BTW what version control is this? Most VC systems I've used are smart enough to only show files as "changed" if the content has changed, not just the timestamp. -- Matt Good From pje at telecommunity.com Thu Apr 5 21:21:26 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Apr 2007 15:21:26 -0400 Subject: [Distutils] How to prevent some .egg-info files being "touched" during a build/install process In-Reply-To: <1175799925.5982.139.camel@nny> References: <625315.78934.qm@web30506.mail.mud.yahoo.com> <625315.78934.qm@web30506.mail.mud.yahoo.com> Message-ID: <5.1.1.6.0.20070405152014.02c43eb0@sparrow.telecommunity.com> At 03:05 PM 4/5/2007 -0400, Matt Good wrote: >I typically add the .egg-info folder to the "ignored" files in version >control since the files in there will be automatically regenerated by >the build process. Unless you have custom files that you put in >the .egg-info folder (which I think is uncommon) you can probably just >leave that folder out of version control too. That's correct. Only projects that explicitly put files under .egg-info need to track them in version control, because all the automatically generated files are regenerated from setup.py whenever any command that needs them is run. From venkatbo at yahoo.com Thu Apr 5 23:28:38 2007 From: venkatbo at yahoo.com (Venkat Bommakanti) Date: Thu, 5 Apr 2007 14:28:38 -0700 (PDT) Subject: [Distutils] How to prevent some .egg-info files being "touched" during a build/install process In-Reply-To: <5.1.1.6.0.20070405152014.02c43eb0@sparrow.telecommunity.com> Message-ID: <20070405212838.71439.qmail@web30507.mail.mud.yahoo.com> Thanks, Matt & Philip. I'll get them out of source control - since those files were deployed in the installed egg, I thought they were needed to be under source control. Just like one can specify temp build & dist dirs to be used in the build/install process, is it possible to have setuptools place these files in a user-defined dir... that'd make it easy to implement the "make clean" equivalent to ensure these files don't show up as being "privtate" files in one's tree. Thanks, /venkat --- "Phillip J. Eby" wrote: > At 03:05 PM 4/5/2007 -0400, Matt Good wrote: > >I typically add the .egg-info folder to the > "ignored" files in version > >control since the files in there will be > automatically regenerated by > >the build process. Unless you have custom files > that you put in > >the .egg-info folder (which I think is uncommon) > you can probably just > >leave that folder out of version control too. > > That's correct. Only projects that explicitly put > files under .egg-info > need to track them in version control, because all > the automatically > generated files are regenerated from setup.py > whenever any command that > needs them is run. > > ____________________________________________________________________________________ Bored stiff? Loosen up... Download and play hundreds of games for free on Yahoo! Games. http://games.yahoo.com/games/front From pje at telecommunity.com Thu Apr 5 23:50:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 05 Apr 2007 17:50:30 -0400 Subject: [Distutils] How to prevent some .egg-info files being "touched" during a build/install process In-Reply-To: <20070405212838.71439.qmail@web30507.mail.mud.yahoo.com> References: <5.1.1.6.0.20070405152014.02c43eb0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070405174912.02f47f20@sparrow.telecommunity.com> At 02:28 PM 4/5/2007 -0700, Venkat Bommakanti wrote: >Thanks, Matt & Philip. > >I'll get them out of source control - since those >files were deployed in the installed egg, I thought >they were needed to be under source control. > >Just like one can specify temp build & dist dirs to >be used in the build/install process, is it possible >to have setuptools place these files in a user-defined >dir... that'd make it easy to implement the >"make clean" equivalent to ensure these files don't >show up as being "privtate" files in one's tree. See the egg_info command options, but please note that "setup.py develop" and "setup.py test" will not work correctly unless the .egg-info directory is in its default location. So, if you can live with that limitation, yes, you can change where the egg-info lives. If not, then the answer is no. From dale-keyword-gmane.42aa79 at codefu.org Sat Apr 7 02:02:32 2007 From: dale-keyword-gmane.42aa79 at codefu.org (Dale Sedivec) Date: Fri, 06 Apr 2007 20:02:32 -0400 Subject: [Distutils] [PATCH] Add Monotone support to setuptools Message-ID: Greetings. Per the setuptools docs, below is patch to add support for the Monotone revision control system[1] to setuptools. I've tested this patch lightly on Linux (Fedora Core 5) with Python 2.4.3 and Monotone 0.33. Patch is against setuptools r54707 (which I think is "head"). Apologies if this isn't the appropriate place for these patches. If that's the case and someone could advise me where they should go (setuptools documentation doesn't seem too specific) I would be grateful. Dale [1] http://monotone.ca/ -- Add support for automatically detecting files under control by Monotone (mtn_revctrl). I also renamed _default_revctrl to svn_cvs_revctrl, so that the naming could be more consistent. Clearly my patches to the documentation were a bit slapdash, simply adding Monotone to the list where appropriate; apologies. Index: setuptools/command/sdist.py =================================================================== --- setuptools/command/sdist.py (revision 54707) +++ setuptools/command/sdist.py (working copy) @@ -1,6 +1,7 @@ from distutils.command.sdist import sdist as _sdist from distutils.util import convert_path import os, re, sys, pkg_resources +import popen2 entities = [ ("<","<"), (">", ">"), (""", '"'), ("'", "'"), @@ -45,7 +46,7 @@ for item in ep.load()(dirname): yield item -def _default_revctrl(dirname=''): +def svn_cvs_revctrl(dirname=''): for path, finder in finders: path = joinpath(dirname,path) if os.path.isfile(path): @@ -53,7 +54,7 @@ if os.path.isfile(path): yield path elif os.path.isdir(path): - for item in _default_revctrl(path): + for item in svn_cvs_revctrl(path): yield item def externals_finder(dirname, filename): @@ -107,20 +108,147 @@ ] +class Monotone (object): + def __init__(self, command, capture_stderr=False): + if capture_stderr: + popen_constructor = popen2.Popen4 + else: + popen_constructor = popen2.Popen3 + mtn_executable = os.environ.get("MTN", "mtn") + popen = self._popen = popen_constructor("%s --xargs -" + % (mtn_executable,)) + popen.tochild.write(command) + popen.tochild.close() + self.readline = popen.fromchild.readline + self._closed = False + def __iter__(self): + return iter(self._popen.fromchild) + def iter_inventory(self): + "Yields valid file names from 'automate inventory' output." + for line in self: + # The specs for this are in monotone's automate.cc. Briefly: + # + # * First column is " " for unchanged, "D"eleted, or + # "R"enamed from + # * Second column is " " for unchanged, "R"enamed to, or + # "A"dded + # * Third column is " " for unchanged, "P"atched, "U"nknown, + # "I"gnored, or "M"issing (from the filesystem) + # + # Then a space, two ID numbers separated by a space, + # another space, and the file name. So we strip the first + # columns, split on space twice, and return the tail end + # of the line which should be path/file. + line = line.rstrip() + if line[0] == " " and line[2] in " P": + yield line[4:].split(" ", 2)[2] + def discard_output(self): + for line in self: + pass + def _close(self): + assert not self._closed + self._popen.fromchild.close() + result = self._popen.wait() + if os.WIFEXITED(result): + self.exit_status = (True, os.WEXITSTATUS(result)) + else: + self.exit_status = (False, result) + self._closed = True + return self.exit_status + def exited_normally(self): + if not self._closed: + self._close() + exited, code = self.exit_status + return exited and code == 0 +def acceptable_mtn_version(): + mtn = Monotone("automate interface_version", capture_stderr=True) + version = mtn.readline().strip() + if not mtn.exited_normally(): + return False + try: + major, minor = version.split(".", 1) + if 1 <= int(major) <= 4: + return True + else: + distutils.log.warn("no support for mtn automate version %r" + % (version,)) + return False + except: + distutils.log.warn("couldn't parse mtn automate version %r" + % (version,)) + return False +def find_mtn_root(a_path): + assert os.path.isabs(a_path) + assert a_path == os.path.normpath(a_path) + if os.path.isdir(os.path.join(a_path, "_MTN")): + return a_path + else: + head, tail = os.path.split(a_path) + if head != a_path: + return find_mtn_root(head) + else: + return None +def get_path_relative_to_mtn_root(mtn_root, abs_path): + common_path = os.path.commonprefix([mtn_root, abs_path]) + return abs_path[len(common_path) + 1:] +def call_with_cwd(temp_cwd, callable, *args, **kwargs): + orig_cwd = os.getcwd() + try: + try: + os.chdir(temp_cwd) + except OSError, e: + distutils.log.warn("error with mtn plug-in: %s" % (str(e),)) + return None + return callable(*args, **kwargs) + finally: + os.chdir(orig_cwd) +def mtn_revctrl(a_path): + abs_path = os.path.normpath(os.path.abspath(a_path)) + mtn_root = find_mtn_root(abs_path) + if mtn_root is None or not acceptable_mtn_version(): + return + mtn_relative_path = get_path_relative_to_mtn_root(mtn_root, abs_path) + if mtn_relative_path: + strip_before = len(mtn_relative_path) + 1 + else: + strip_before = 0 + mtn = call_with_cwd(abs_path, Monotone, "automate inventory") + if not mtn: + return + for file in mtn.iter_inventory(): + if (file.startswith(mtn_relative_path) + and file.rstrip("/") != mtn_relative_path): + yield os.path.join(a_path, file[strip_before:]) + if not mtn.exited_normally(): + distutils.log.warn("error running mtn: exited=%r status=%r" + % mtn.exit_status) + + + + + + + + + + + + + class sdist(_sdist): """Smart sdist that finds anything supported by revision control""" Index: setup.py =================================================================== --- setup.py (revision 54707) +++ setup.py (working copy) @@ -70,7 +70,8 @@ % sys.version[:3] ], "setuptools.file_finders": - ["svn_cvs = setuptools.command.sdist:_default_revctrl"], + ["svn_cvs = setuptools.command.sdist:svn_cvs_revctrl", + "mtn = setuptools.command.sdist:mtn_revctrl"], "setuptools.installation": Index: setuptools.txt =================================================================== --- setuptools.txt (revision 54707) +++ setuptools.txt (working copy) @@ -261,10 +261,10 @@ ``include_package_data`` If set to ``True``, this tells ``setuptools`` to automatically include any - data files it finds inside your package directories, that are either under - CVS or Subversion control, or which are specified by your ``MANIFEST.in`` - file. For more information, see the section below on `Including Data - Files`_. + data files it finds inside your package directories, that are under CVS, + Subversion, or Monotone control, or data files which are specified by your + ``MANIFEST.in`` file. For more information, see the section below on + `Including Data Files`_. ``exclude_package_data`` A dictionary mapping package names to lists of glob patterns that should @@ -745,10 +745,10 @@ ) This tells setuptools to install any data files it finds in your packages. The -data files must be under CVS or Subversion control, or else they must be -specified via the distutils' ``MANIFEST.in`` file. (They can also be tracked -by another revision control system, using an appropriate plugin. See the -section below on `Adding Support for Other Revision Control Systems`_ for +data files must be under CVS, Subversion, or Monotone control, or else they +must be specified via the distutils' ``MANIFEST.in`` file. (They can also be +tracked by another revision control system, using an appropriate plugin. See +the section below on `Adding Support for Other Revision Control Systems`_ for information on how to write such plugins.) If you want finer-grained control over what files are included (for example, if @@ -1429,17 +1429,17 @@ ------------------------------- ``setuptools`` enhances the distutils' default algorithm for source file -selection, so that all files managed by CVS or Subversion in your project tree -are included in any source distribution you build. This is a big improvement -over having to manually write a ``MANIFEST.in`` file and try to keep it in -sync with your project. So, if you are using CVS or Subversion, and your -source distributions only need to include files that you're tracking in -revision control, don't create a a ``MANIFEST.in`` file for your project. -(And, if you already have one, you might consider deleting it the next time -you would otherwise have to change it.) +selection, so that all files managed by CVS, Subversion, or Monotone in your +project tree are included in any source distribution you build. This is a big +improvement over having to manually write a ``MANIFEST.in`` file and try to +keep it in sync with your project. So, if you are using CVS, Subversion, or +Monotone, and your source distributions only need to include files that you're +tracking in revision control, don't create a a ``MANIFEST.in`` file for your +project. (And, if you already have one, you might consider deleting it the +next time you would otherwise have to change it.) -(NOTE: other revision control systems besides CVS and Subversion can be -supported using plugins; see the section below on `Adding Support for Other +(NOTE: other revision control systems besides CVS, Subversion, and Monotone can +be supported using plugins; see the section below on `Adding Support for Other Revision Control Systems`_ for information on how to write such plugins.) If you need to include automatically generated files, or files that are kept in @@ -2495,9 +2495,9 @@ ------------------------------------------------- If you would like to create a plugin for ``setuptools`` to find files in other -source control systems besides CVS and Subversion, you can do so by adding an -entry point to the ``setuptools.file_finders`` group. The entry point should -be a function accepting a single directory name, and should yield +source control systems besides CVS, Subversion, and Monotone, you can do so by +adding an entry point to the ``setuptools.file_finders`` group. The entry +point should be a function accepting a single directory name, and should yield all the filenames within that directory (and any subdirectories thereof) that are under revision control. Index: setuptools.egg-info/entry_points.txt =================================================================== --- setuptools.egg-info/entry_points.txt (revision 54707) +++ setuptools.egg-info/entry_points.txt (working copy) @@ -35,7 +35,8 @@ easy_install-2.3 = setuptools.command.easy_install:main [setuptools.file_finders] -svn_cvs = setuptools.command.sdist:_default_revctrl +svn_cvs = setuptools.command.sdist:svn_cvs_revctrl +mtn = setuptools.command.sdist:mtn_revctrl [distutils.setup_keywords] dependency_links = setuptools.dist:assert_string_list From pje at telecommunity.com Sat Apr 7 04:57:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 06 Apr 2007 22:57:12 -0400 Subject: [Distutils] [PATCH] Add Monotone support to setuptools In-Reply-To: Message-ID: <5.1.1.6.0.20070406225551.02dda9b0@sparrow.telecommunity.com> At 08:02 PM 4/6/2007 -0400, Dale Sedivec wrote: > Greetings. Per the setuptools docs, below is patch to add >support for the Monotone revision control system[1] to setuptools. May I suggest, that instead of distributing this as a patch, you simply create a package that contains this support? Then, anyone using Monotone can simply easy_install your package. I would prefer not to have to support this code in the main setuptools distribution. From pje at telecommunity.com Sat Apr 7 07:47:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Apr 2007 01:47:59 -0400 Subject: [Distutils] [PATCH] Add Monotone support to setuptools In-Reply-To: <20070407031142.GA21926@morgase.caliginous.net> References: <5.1.1.6.0.20070406225551.02dda9b0@sparrow.telecommunity.com> <5.1.1.6.0.20070406225551.02dda9b0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070407014429.044e4ea0@sparrow.telecommunity.com> At 11:11 PM 4/6/2007 -0400, Dale Sedivec wrote: >On Fri, Apr 06, 2007 at 10:57:12PM -0400, Phillip J. Eby wrote: > > At 08:02 PM 4/6/2007 -0400, Dale Sedivec wrote: > > > Greetings. Per the setuptools docs, below is patch to add > > >support for the Monotone revision control system[1] to setuptools. > > > > May I suggest, that instead of distributing this as a patch, you simply > > create a package that contains this support? Then, anyone using Monotone > > can simply easy_install your package. I would prefer not to have to > > support this code in the main setuptools distribution. > > That was actually my original intent, but then I saw the >setuptools documentation suggested "submitting a patch to the >setuptools.command.sdist module," Oops -- I missed taking that out when I added a plugin interface. (Originally, patching was the only way to add support for a new source control system.) Thanks for pointing this out; I'll fix the docs pronto. :) From pje at telecommunity.com Sat Apr 7 18:44:56 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 07 Apr 2007 12:44:56 -0400 Subject: [Distutils] Time for a static cheeseshop mirror for easy_install? Message-ID: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> For what easy_install does, there really isn't any dynamic API usage, so a static mirror for easy_install could take a good bit of load off the cheeseshop. I don't know whether this will actually solve any problems the cheeseshop itself is having; it may be that ill-behaved web spiders are at fault, or something else altogether. However, since the downtime mostly creates issues for people using easy_install, creating a solution for those people certainly seems worthwhile. Since easy_install was designed to be able to use simple directory indexes and HTML pages as a package index, it should be possible to create a simple directory tree of HTML pages, using PyPI's public XML-RPC API. The mirror could use PyPI's RSS feed to know when a package's information is out of date, although I'm not sure that the RSS includes all modifications, such as when packages are deleted, releases are hidden, files uploaded, etc. However, assuming that there's a scalable way to receive change notifications, it should be straightforward to implement a mirror script for easy_install, and have it run on one or more volunteered hosts, perhaps with round-robin DNS (maybe easy-install.python.org?) I'll be happy to assist anyone who wants to work on this, including updates to easy_install itself, of course. I'd actually be hacking on this now, if the cheeseshop weren't down (i.e., I can't download any XML-RPC data to do prototyping at the moment!) From g.brandl at gmx.net Sun Apr 8 14:29:58 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 08 Apr 2007 14:29:58 +0200 Subject: [Distutils] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> Message-ID: Phillip J. Eby schrieb: > For what easy_install does, there really isn't any dynamic API usage, so a > static mirror for easy_install could take a good bit of load off the > cheeseshop. > > I don't know whether this will actually solve any problems the cheeseshop > itself is having; it may be that ill-behaved web spiders are at fault, or > something else altogether. However, since the downtime mostly creates > issues for people using easy_install, creating a solution for those people > certainly seems worthwhile. > > Since easy_install was designed to be able to use simple directory indexes > and HTML pages as a package index, it should be possible to create a simple > directory tree of HTML pages, using PyPI's public XML-RPC API. The mirror > could use PyPI's RSS feed to know when a package's information is out of > date, although I'm not sure that the RSS includes all modifications, such > as when packages are deleted, releases are hidden, files uploaded, etc. > > However, assuming that there's a scalable way to receive change > notifications, it should be straightforward to implement a mirror script > for easy_install, and have it run on one or more volunteered hosts, perhaps > with round-robin DNS (maybe easy-install.python.org?) The German Python community is willing to contribute a host for PyPI mirroring. Just contact me as soon as some solution is found. Georg From amk at amk.ca Mon Apr 9 16:46:30 2007 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 9 Apr 2007 10:46:30 -0400 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> Message-ID: <20070409144630.GA10277@localhost.localdomain> On Sat, Apr 07, 2007 at 12:44:56PM -0400, Phillip J. Eby wrote: > I don't know whether this will actually solve any problems the cheeseshop > itself is having; it may be that ill-behaved web spiders are at fault, or > something else altogether. In this recent case, two different spiders were crawling the wiki very quickly, the machine's load average was in the 70s, and the out-of-memory killer was killing off PostgreSQL processes. I don't think the load caused from people running easy_install is especially high -- it's certainly not a source of problems -- but making static pages would still be good to make mirroring the package archive more useful. Right now people could mirror http://cheeseshop.python.org/packages/, but there's nothing there for easy_install or for human readers; it's just a tree of package directories. --amk From jim at zope.com Mon Apr 9 17:06:56 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Apr 2007 11:06:56 -0400 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <20070409144630.GA10277@localhost.localdomain> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> <20070409144630.GA10277@localhost.localdomain> Message-ID: On Apr 9, 2007, at 10:46 AM, A.M. Kuchling wrote: ... > Right now people could mirror > http://cheeseshop.python.org/packages/, but there's nothing there for > easy_install Is there any reason why easy_install shouldn't look there? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Mon Apr 9 17:32:20 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 09 Apr 2007 17:32:20 +0200 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> <20070409144630.GA10277@localhost.localdomain> Message-ID: <461A5C84.8010101@v.loewis.de> Jim Fulton schrieb: > On Apr 9, 2007, at 10:46 AM, A.M. Kuchling wrote: > ... >> Right now people could mirror >> http://cheeseshop.python.org/packages/, but there's nothing there for >> easy_install > > Is there any reason why easy_install shouldn't look there? It's only files, no descriptions. Also, some authors don't have their files on /packages, but at their own servers - i.e. they use the Cheeseshop just as a Python package index, not as a comprehensive package archive network. Regards, Martin From jim at zope.com Mon Apr 9 17:35:34 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 9 Apr 2007 11:35:34 -0400 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <461A5C84.8010101@v.loewis.de> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> <20070409144630.GA10277@localhost.localdomain> <461A5C84.8010101@v.loewis.de> Message-ID: <56EE93DB-F8DB-476A-AD8A-ADE48E02C858@zope.com> On Apr 9, 2007, at 11:32 AM, Martin v. L?wis wrote: > Jim Fulton schrieb: >> On Apr 9, 2007, at 10:46 AM, A.M. Kuchling wrote: >> ... >>> Right now people could mirror >>> http://cheeseshop.python.org/packages/, but there's nothing there >>> for >>> easy_install >> >> Is there any reason why easy_install shouldn't look there? > > It's only files, no descriptions. > > Also, some authors don't have their files on /packages, but at their > own servers - i.e. they use the Cheeseshop just as a Python > package index, not as a comprehensive package archive network. Sorry, I wasn't clear. I wasn't suggesting that easy_install should look here instead of where it already looks, but this seems like a good and cheap place to look especially when a specific version number is given. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From matt at matt-good.net Mon Apr 9 17:56:25 2007 From: matt at matt-good.net (Matt Good) Date: Mon, 09 Apr 2007 11:56:25 -0400 Subject: [Distutils] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> Message-ID: <1176134186.5675.4.camel@nny> On Sat, 2007-04-07 at 12:44 -0400, Phillip J. Eby wrote: > For what easy_install does, there really isn't any dynamic API usage, so a > static mirror for easy_install could take a good bit of load off the > cheeseshop. Jon Rosebaugh (aka Chairos) actually started working on this a couple days ago during a Cheeseshop downtime. I mentioned this post to him, so he may join the list to follow up, but you can reach him on #python.web to check on this. -- Matt Good From pje at telecommunity.com Mon Apr 9 20:24:02 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Apr 2007 14:24:02 -0400 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <20070409144630.GA10277@localhost.localdomain> References: <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070409141838.02dc73c8@sparrow.telecommunity.com> At 10:46 AM 4/9/2007 -0400, A.M. Kuchling wrote: >On Sat, Apr 07, 2007 at 12:44:56PM -0400, Phillip J. Eby wrote: > > I don't know whether this will actually solve any problems the cheeseshop > > itself is having; it may be that ill-behaved web spiders are at fault, or > > something else altogether. > >In this recent case, two different spiders were crawling the wiki very >quickly, the machine's load average was in the 70s, and the >out-of-memory killer was killing off PostgreSQL processes. > >I don't think the load caused from people running easy_install is >especially high -- it's certainly not a source of problems -- but >making static pages would still be good to make mirroring the package >archive more useful. Right now people could mirror >http://cheeseshop.python.org/packages/, but there's nothing there for >easy_install or for human readers; it's just a tree of package >directories. Hm. Well, actually, if that directory structure were something I could code to, easy_install could sure as heck be *made* to use it. The only thing easy_install couldn't get from it was external links to downloads, SVN versions, etc. Notice, for example, that if you use "easy_install -f http://cheeseshop.python.org/packages/source/s/simple_json/ simple_json", easy_install won't look at the main package index, but just download directly. So an automated form of that calculation could easily be added to easy_install. What I had in mind for an easy_install mirror, however, was a script that would just create a /packagename/index.html file with links gathered from all versions of a package on the original Cheeseshop, and with packagename generated as a setuptools "safe name" in lower case. This pattern would allow easy_install to find every possible relevant link in just one (static) web hit. From pje at telecommunity.com Mon Apr 9 20:30:17 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 09 Apr 2007 14:30:17 -0400 Subject: [Distutils] [Catalog-sig] Time for a static cheeseshop mirror for easy_install? In-Reply-To: <56EE93DB-F8DB-476A-AD8A-ADE48E02C858@zope.com> References: <461A5C84.8010101@v.loewis.de> <5.1.1.6.0.20070407122140.02c67950@sparrow.telecommunity.com> <20070409144630.GA10277@localhost.localdomain> <461A5C84.8010101@v.loewis.de> Message-ID: <5.1.1.6.0.20070409142441.0503fa78@sparrow.telecommunity.com> At 11:35 AM 4/9/2007 -0400, Jim Fulton wrote: >On Apr 9, 2007, at 11:32 AM, Martin v. L?wis wrote: > > > Jim Fulton schrieb: > >> On Apr 9, 2007, at 10:46 AM, A.M. Kuchling wrote: > >> ... > >>> Right now people could mirror > >>> http://cheeseshop.python.org/packages/, but there's nothing there > >>> for > >>> easy_install > >> > >> Is there any reason why easy_install shouldn't look there? > > > > It's only files, no descriptions. > > > > Also, some authors don't have their files on /packages, but at their > > own servers - i.e. they use the Cheeseshop just as a Python > > package index, not as a comprehensive package archive network. > >Sorry, I wasn't clear. I wasn't suggesting that easy_install should >look here instead of where it already looks, but this seems like a >good and cheap place to look especially when a specific version >number is given. Actually, version number doesn't help here; it's Python version, and source vs. binary that are the relevant distinctions. The download directories include files for all versions of a given project, so we would want to hit these three locations: http://cheeseshop.python.org/packages/*pyver*/p/projectname/ http://cheeseshop.python.org/packages/any/p/projectname/ http://cheeseshop.python.org/packages/source/p/projectname/ And we would then have all the downloadable-from-cheeseshop links. (I thought we could skip the "any", but some win32 exe's are classed as "any" Python version, and easy_install supports win32 exe's.) This is more web hits than is currently required to obtain similar information, but on the plus side they are efficient hits for the server. They also allow access to *all* versions of a package that are downloadable, whether they are "hidden" or not. On the minus side, there is no way to find externally-hosted files or SVN links, which means we would still have to always hit the dynamic page to know if we had found everything relevant. From chairos at gmail.com Tue Apr 10 08:10:21 2007 From: chairos at gmail.com (Jon Rosebaugh) Date: Tue, 10 Apr 2007 01:10:21 -0500 Subject: [Distutils] My Cheeseshop mirror Message-ID: As Matt Good mentioned in the "Time for a static cheeseshop mirror for easy_install?", I started working on a cheeseshop mirror during one of the downtimes last week. It's now up and running in a limited capacity: http://cheeseboard.inklesspen.com/ The immediate goal of the "cheeseboard" is to enable people to install setuptools, Pylons, Mako, and other packages in the Pylons family when the cheeseshop is down, without having to jump through more than one or two hoops. It's a dynamic webapp written in Pylons, but if the load gets to be too big, it's easy to cache almost all of it. The files themselves are stored on Amazon S3; I'll have to see if that's cost-effective for this. Currently it is a manually-updated partial mirror. You have to add a 'package', which consists of the name and optionally SVN info and homepage, and then feed it the cheeseshop url for a specific release, such as . It then copies all eggs, tarballs and zip files to S3 and enters the relevant info in the database. I've written it in a manner that should be usable by other people, but the SVN repository is not currently world-readable, and I haven't gotten around to uploading it to the cheeseshop yet. But if there's interest, I could do these things. In the future, I expect to add functionality for adding a package and all its dependencies automatically. I don't expect to be adding functionality for mirroring the complete cheeseshop, though, partially due to not having a good idea for how to do that automatically. Jon From antoine.pitrou at wengo.com Tue Apr 10 14:14:26 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Tue, 10 Apr 2007 14:14:26 +0200 Subject: [Distutils] spurious warning when multiple ez_setup instances are present Message-ID: <1176207266.6674.7.camel@antoine-ubuntu> Hi, I occasionally get this warning when using a setuptools-enabled setup.py from a third-party package: /usr/lib/python2.5/site-packages/setuptools/command/sdist.py:3: UserWarning: Module ez_setup was already imported from /home/antoine/nose/config-file/ez_setup.pyc, but /home/antoine/XXX is being added to sys.path It seems to me that this warning conveys no useful information (yes, several independent packages can have their own copy of ez_setup.py... so what?), and that it would be better to remove it. Regards Antoine. From antoine.pitrou at wengo.com Tue Apr 10 13:58:42 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Tue, 10 Apr 2007 13:58:42 +0200 Subject: [Distutils] spurious warning when multiple ez_setup instances are present Message-ID: <1176206322.6674.5.camel@antoine-ubuntu> Hi, I occasionally get this warning when using a setuptools-enabled setup.py from a third-party package: /usr/lib/python2.5/site-packages/setuptools/command/sdist.py:3: UserWarning: Module ez_setup was already imported from /home/antoine/nose/config-file/ez_setup.pyc, but /home/antoine/XXX is being added to sys.path It seems to me that this warning conveys no useful information (yes, several independent packages can have their own copy of ez_setup.py... so what?), and so that it would be better to remove it. Regards Antoine. From pje at telecommunity.com Tue Apr 10 18:19:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 10 Apr 2007 12:19:44 -0400 Subject: [Distutils] spurious warning when multiple ez_setup instances are present In-Reply-To: <1176207266.6674.7.camel@antoine-ubuntu> Message-ID: <5.1.1.6.0.20070410121216.02c00458@sparrow.telecommunity.com> At 02:14 PM 4/10/2007 +0200, Antoine Pitrou wrote: >Hi, > >I occasionally get this warning when using a setuptools-enabled setup.py >from a third-party package: > >/usr/lib/python2.5/site-packages/setuptools/command/sdist.py:3: >UserWarning: Module ez_setup was already imported >from /home/antoine/nose/config-file/ez_setup.pyc, but /home/antoine/XXX >is being added to sys.path > >It seems to me that this warning conveys no useful information (yes, >several independent packages can have their own copy of ez_setup.py... >so what?), and that it would be better to remove it. This isn't a warning about ez_setup specifically - it's a warning that occurs whenever an imported module will be shadowed by an egg being added to sys.path. However, this should only happen for modules and packages specifically included in a project... and ez_setup should *not* be included in a project's installed modules or packages. Further investigation suggests that the issue is actually find_packages(), which includes 'ez_setup' as a package if run against the project root without 'exclude=["ez_setup"]'. In other words, you can get rid of this message by changing the find_packages() call being used in the package at /home/antoine/XXX. I'll also change find_packages() to ensure that it always excludes ez_setup in future, since if someone wants to force ez_setup to be included in the installation (as opposed to the source distribution), they can always explicitly add it to their package list. I am somewhat concerned as to whether this will still cause ez_setup to be packaged with a source distribution in the case where someone is not using revision control; I'll have to test that case to be sure. From robinbryce at gmail.com Tue Apr 10 20:01:50 2007 From: robinbryce at gmail.com (Robin Bryce) Date: Tue, 10 Apr 2007 19:01:50 +0100 Subject: [Distutils] Buildout site-dirs and develop eggs Message-ID: Hi, When buildout installs develop eggs how does it setup --site-dirs ? I've tried having a rummage in buildout.py but I don't understand what the Buildout._develop method is doing; It appears to be preparing a environ dict - presumably in preparation for execing the moral equivalent of 'python setup.py develop' - but it does not pass this to zc.buildout.easy_install.develop. It _does_ pass in dest. I can see this comes from buildout['buildout']['develop-eggs-directory']; but it doesn't appear to do anything special to propagate this to --site-dir. I have sometimes works sometimes doesn't symptoms (see below) - Is there current working dir nonsense going on here ? The relevant versions are: zc.buildout.easy_install.picked: zc.buildout = 1.0.0b23 zc.buildout.easy_install.picked: setuptools = 0.6c5 I'm getting this error from running "buildout -vvv install": buildout: Develop: /home/robin/devel-asycamore/recipes zc.buildout.easy_install: in: /home/robin/devel-asycamore/recipes ['/tmp/tmpzqO9yC', '-q', 'develop', '-mxN', '-d', '/home/robin/devel-asycamore/./tmpxiPaPrbuild'] buildout: Develop: /home/robin/devel-asycamore/asycamore-trunk zc.buildout.easy_install: in: /home/robin/devel-asycamore/asycamore-trunk ['/tmp/tmp06R7ze', '-q', 'develop', '-mxN', '-d', '/home/robin/devel-asycamore/./tmppOiykWbuild'] error: /home/robin/devel-servers (in --site-dirs) is not on sys.path While: Installing Processing develop directory /home/robin/devel-asycamore/asycamore-trunk An internal error occured ... This is my first pass through working with zc.buildout and I think my buildout.cfg suspect. I have a recipe that does an svn co/up, a part that 'installs' sources using it, and dependent part that builds a release egg. At the very top I'm also listing the 'part' produced svn checkout as a 'develop' egg. As develop eggs get built before non develop eggs I dont think this will work in a single pass but even so the --site-dirs error was not what I was expecting. Is there a 'right way' tm to have a buildout install a develop link based on a part produced checkout ? I got some hints from this thread: http://mail.python.org/pipermail/distutils-sig/2007-January/007160.html but didn't see any clear recommendations. My buildout - such as it is - can be found here: http://svn.wiretooth.com/svn/open/asycamore.developer/trunk/ Cheers, Robin From robinbryce at gmail.com Tue Apr 10 21:02:51 2007 From: robinbryce at gmail.com (Robin Bryce) Date: Tue, 10 Apr 2007 20:02:51 +0100 Subject: [Distutils] Buildout site-dirs and develop eggs In-Reply-To: References: Message-ID: Ah, woops. I had an unhelpful setup.cfg in one of the source checkouts I was tracking. And the way zc.buildout deals with develop targets being dependent on parts is kind of neat. If the target is missing, it just goes a head with the parts anyway. If you run the buildout a second time you get the develop links to the sources installed by the previous run. And as they are 'develop' links it doesn't matter if the subsequent part steps do update or un-install/install. Nice one :) Robin On 10/04/07, Robin Bryce wrote: > Hi, > > When buildout installs develop eggs how does it setup --site-dirs ? > > I've tried having a rummage in buildout.py but I don't understand what > the Buildout._develop method is doing; It appears to be preparing a > environ dict - presumably in preparation for execing the moral > equivalent of 'python setup.py develop' - but it does not pass this to > zc.buildout.easy_install.develop. It _does_ pass in dest. I can see > this comes from buildout['buildout']['develop-eggs-directory']; but it > doesn't appear to do anything special to propagate this to --site-dir. > > I have sometimes works sometimes doesn't symptoms (see below) - Is > there current working dir nonsense going on here ? > > The relevant versions are: > zc.buildout.easy_install.picked: zc.buildout = 1.0.0b23 > zc.buildout.easy_install.picked: setuptools = 0.6c5 > > I'm getting this error from running "buildout -vvv install": > > buildout: Develop: /home/robin/devel-asycamore/recipes > zc.buildout.easy_install: in: /home/robin/devel-asycamore/recipes > ['/tmp/tmpzqO9yC', '-q', 'develop', '-mxN', '-d', > '/home/robin/devel-asycamore/./tmpxiPaPrbuild'] > buildout: Develop: /home/robin/devel-asycamore/asycamore-trunk > zc.buildout.easy_install: in: /home/robin/devel-asycamore/asycamore-trunk > ['/tmp/tmp06R7ze', '-q', 'develop', '-mxN', '-d', > '/home/robin/devel-asycamore/./tmppOiykWbuild'] > error: /home/robin/devel-servers (in --site-dirs) is not on sys.path > While: > Installing > Processing develop directory /home/robin/devel-asycamore/asycamore-trunk > > An internal error occured ... > > > This is my first pass through working with zc.buildout and I think my > buildout.cfg suspect. I have a recipe that does an svn co/up, a part > that 'installs' sources using it, and dependent part that builds a > release egg. At the very top I'm also listing the 'part' produced svn > checkout as a 'develop' egg. As develop eggs get built before non > develop eggs I dont think this will work in a single pass but even so > the --site-dirs error was not what I was expecting. > > Is there a 'right way' tm to have a buildout install a develop link > based on a part produced checkout ? > > I got some hints from this thread: > http://mail.python.org/pipermail/distutils-sig/2007-January/007160.html > but didn't see any clear recommendations. > > My buildout - such as it is - can be found here: > http://svn.wiretooth.com/svn/open/asycamore.developer/trunk/ > > Cheers, > Robin > From antoine.pitrou at wengo.com Wed Apr 11 12:03:13 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Wed, 11 Apr 2007 12:03:13 +0200 Subject: [Distutils] spurious warning when multiple ez_setup instances are present In-Reply-To: <5.1.1.6.0.20070410121216.02c00458@sparrow.telecommunity.com> References: <5.1.1.6.0.20070410121216.02c00458@sparrow.telecommunity.com> Message-ID: <1176285793.6818.6.camel@antoine-ubuntu> Hi, Le mardi 10 avril 2007 ? 12:19 -0400, Phillip J. Eby a ?crit : > This isn't a warning about ez_setup specifically - it's a warning that > occurs whenever an imported module will be shadowed by an egg being added > to sys.path. Thanks, I see. > Further investigation suggests that the issue is actually find_packages(), > which includes 'ez_setup' as a package if run against the project root > without 'exclude=["ez_setup"]'. In other words, you can get rid of this > message by changing the find_packages() call being used in the package at > /home/antoine/XXX. Well, find_packages() isn't used at all. The situation is that : - /home/antoine/nose/config-file/ez_setup.py is an ez_setup.py bundled with nose (see http://python-nose.googlecode.com/svn/trunk/ for the nose SVN) - /home/antoine/XXX is another package which does an SVN external reference to the ez_setup SVN (svn://svn.eby-sarna.com/svnroot/ez_setup) That's why two different ez_setup versions end up "conflicting". It seems to me that bundling ez_setup as a convenience for the package user is a common idiom, or is it deprecated? Thanks Antoine. From antoine.pitrou at wengo.com Wed Apr 11 12:07:28 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Wed, 11 Apr 2007 12:07:28 +0200 Subject: [Distutils] spurious warning when multiple ez_setup instances are present In-Reply-To: <1176285793.6818.6.camel@antoine-ubuntu> References: <5.1.1.6.0.20070410121216.02c00458@sparrow.telecommunity.com> <1176285793.6818.6.camel@antoine-ubuntu> Message-ID: <1176286048.6818.12.camel@antoine-ubuntu> Le mercredi 11 avril 2007 ? 12:03 +0200, Antoine Pitrou a ?crit : > Well, find_packages() isn't used at all. Oops, sorry, I'm not very awake :-( Yes of course find_packages() is used in the setup.py file. Antoine. From pje at telecommunity.com Wed Apr 11 18:17:01 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 11 Apr 2007 12:17:01 -0400 Subject: [Distutils] spurious warning when multiple ez_setup instances are present In-Reply-To: <1176286048.6818.12.camel@antoine-ubuntu> References: <1176285793.6818.6.camel@antoine-ubuntu> <5.1.1.6.0.20070410121216.02c00458@sparrow.telecommunity.com> <1176285793.6818.6.camel@antoine-ubuntu> Message-ID: <5.1.1.6.0.20070411121555.02ace660@sparrow.telecommunity.com> At 12:07 PM 4/11/2007 +0200, Antoine Pitrou wrote: >Le mercredi 11 avril 2007 ? 12:03 +0200, Antoine Pitrou a ?crit : > > Well, find_packages() isn't used at all. > >Oops, sorry, I'm not very awake :-( >Yes of course find_packages() is used in the setup.py file. right, and if you make it "find_packages(exclude=['ez_setup'])", it will stop the warning. In a future version, ez_setup will be excluded automatically. From murray at muzmo.com Thu Apr 12 11:17:18 2007 From: murray at muzmo.com (Friends system) Date: Thu, 12 Apr 2007 01:17:18 -0800 Subject: [Distutils] Please reply to confirm:1176369433.1060_11866.navsta message and allow delivery Message-ID: <1176369438_4371@mail.cerusindustrial.com> This message was automatically generated for Murray : The email system that we are using only accept emails from email addresses on personalized 'Lists of Friends' and you are not currently included on my 'List of Friends'. Your ommission from the list is probably due to an oversight on my part. We have added this new mechanism to reduce the amount of unwanted email by attempting to verify that the sender is a human. In the meantime, we are regularly adding new addresses to the 'List of Friends' If you know that I would really want to hear from you: Please reply to this message without changing the subject line, or even adding any text. I will then receive your email and be able to add you to my 'List of Friends' so that, hopefully*, you will not be bothered by this automated reply again, and I will receive your all your emails in the future. * If you have multiple email addresses that you would like me to add to my 'List of Friends', then please send me an email from each account. Thanks, Murray For more information see http://netwinsite.com/surgemail/friends.htm From benji at zope.com Thu Apr 12 20:34:37 2007 From: benji at zope.com (Benji York) Date: Thu, 12 Apr 2007 14:34:37 -0400 Subject: [Distutils] setuptools and packages with dots Message-ID: <461E7BBD.1050205@zope.com> While working on a package I decided to do some old-school revision control and make a copy of the directory. I recursively copied the package to PACKAGENAME.bak1. I then ran setup.py develop (indirectly though zc.buildout) and received an intriguing error message. Here are reproduction steps that don't use buildout. % mkdir foo % cd foo % mkdir bad.dir % touch bad.dir/__init__.py create setup.py as such: from setuptools import setup, find_packages setup( name = 'test', packages = find_packages('.'), install_requires = ['setuptools'] ) % python setup.py develop running develop running egg_info writing requirements to test.egg-info/requires.txt writing test.egg-info/PKG-INFO writing top-level names to test.egg-info/top_level.txt writing dependency_links to test.egg-info/dependency_links.txt error: package directory 'bad/dir' does not exist It looks like somewhere in setuptools paths are being normalized to package names and back. The dot doesn't survive the round trip. -- Benji York Senior Software Engineer Zope Corporation From pje at telecommunity.com Thu Apr 12 21:36:07 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 12 Apr 2007 15:36:07 -0400 Subject: [Distutils] setuptools and packages with dots In-Reply-To: <461E7BBD.1050205@zope.com> Message-ID: <5.1.1.6.0.20070412153423.02f96448@sparrow.telecommunity.com> At 02:34 PM 4/12/2007 -0400, Benji York wrote: >% python setup.py develop >running develop >running egg_info >writing requirements to test.egg-info/requires.txt >writing test.egg-info/PKG-INFO >writing top-level names to test.egg-info/top_level.txt >writing dependency_links to test.egg-info/dependency_links.txt >error: package directory 'bad/dir' does not exist > >It looks like somewhere in setuptools paths are being normalized to >package names and back. The dot doesn't survive the round trip. Er... so? :) From benji at zope.com Thu Apr 12 21:53:30 2007 From: benji at zope.com (Benji York) Date: Thu, 12 Apr 2007 15:53:30 -0400 Subject: [Distutils] setuptools and packages with dots In-Reply-To: <5.1.1.6.0.20070412153423.02f96448@sparrow.telecommunity.com> References: <5.1.1.6.0.20070412153423.02f96448@sparrow.telecommunity.com> Message-ID: <461E8E3A.6020705@zope.com> Phillip J. Eby wrote: > At 02:34 PM 4/12/2007 -0400, Benji York wrote: >> % python setup.py develop >> running develop >> running egg_info >> writing requirements to test.egg-info/requires.txt >> writing test.egg-info/PKG-INFO >> writing top-level names to test.egg-info/top_level.txt >> writing dependency_links to test.egg-info/dependency_links.txt >> error: package directory 'bad/dir' does not exist >> >> It looks like somewhere in setuptools paths are being normalized to >> package names and back. The dot doesn't survive the round trip. > > Er... so? :) Perhaps directories who's names are not valid Python package names should be ignored. -- Benji York Senior Software Engineer Zope Corporation From pje at telecommunity.com Fri Apr 13 02:18:41 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 12 Apr 2007 20:18:41 -0400 Subject: [Distutils] setuptools and packages with dots In-Reply-To: <461E8E3A.6020705@zope.com> References: <5.1.1.6.0.20070412153423.02f96448@sparrow.telecommunity.com> <5.1.1.6.0.20070412153423.02f96448@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070412201656.02e79ea0@sparrow.telecommunity.com> At 03:53 PM 4/12/2007 -0400, Benji York wrote: >Phillip J. Eby wrote: >>At 02:34 PM 4/12/2007 -0400, Benji York wrote: >>>% python setup.py develop >>>running develop >>>running egg_info >>>writing requirements to test.egg-info/requires.txt >>>writing test.egg-info/PKG-INFO >>>writing top-level names to test.egg-info/top_level.txt >>>writing dependency_links to test.egg-info/dependency_links.txt >>>error: package directory 'bad/dir' does not exist >>> >>>It looks like somewhere in setuptools paths are being normalized to >>>package names and back. The dot doesn't survive the round trip. >>Er... so? :) > >Perhaps directories who's names are not valid Python package names should >be ignored. Ah... so you're saying find_packages() is broken. I suppose I agree with that, to a point. From gentoodev at gmail.com Fri Apr 13 19:24:16 2007 From: gentoodev at gmail.com (Rob) Date: Fri, 13 Apr 2007 10:24:16 -0700 Subject: [Distutils] Determining system packages Message-ID: Hi, I'm trying to determine which packages are "system" packages as opposed to development or regular packages for yolk[1]. An example of what I mean would be wsgiref which comes included with Python 2.5. The closest thing I can find that would help in a pkg_resources Distribution object is the precedence attribute, but system and development seem to be the same and the precedence would only help if there were more than one in the environment. If I use get_python_lib from distutils.sysconfig and remove the trailing "site-packages" from the reult and check a distribution's location attribute to see if its installed there, will it be reliable or might system packages be in other locations? Thanks, Rob [1] http://tools.assembla.com/yolk From pje at telecommunity.com Fri Apr 13 20:32:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 13 Apr 2007 14:32:44 -0400 Subject: [Distutils] Determining system packages In-Reply-To: Message-ID: <5.1.1.6.0.20070413141914.02b139c0@sparrow.telecommunity.com> At 10:24 AM 4/13/2007 -0700, Rob wrote: >Hi, > >I'm trying to determine which packages are "system" packages as >opposed to development or regular packages for yolk[1]. > >An example of what I mean would be wsgiref which comes included with >Python 2.5. > >The closest thing I can find that would help in a pkg_resources >Distribution object is the precedence attribute, but system and >development seem to be the same and the precedence would only help if >there were more than one in the environment. > >If I use get_python_lib from distutils.sysconfig and remove the >trailing "site-packages" from the reult You mean 'get_python_lib(standard_lib=1)'? > and check a distribution'slocation attribute to see if its installed > there, will it be reliable >or might system packages be in other locations? System packages are just develop eggs installed in "system" locations. System locations may be Python-defined or OS-defined. For example, some distros have non-standard directory layouts, such that there is both a "/usr/lib/python2.x/site-packages" and a "/usr/local/lib/python2.x/site-packages", and *either* could contain system packages. There is a trick you could use to identify "develop" distributions, though... if there is a "SOURCES.txt" metadata file associated with an egg, you can look for a setup.py in a parent directory whose directory layout corresponds to that described by the contents of SOURCES.txt. This could *potentially* produce a false positive, though, if the package in question puts all its modules in the project root and doesn't have any other files besides setup.py listed in the project root. (The long-term plan for 0.7 is that packages that are actually "installed" will have some kind of installation manifest ala PEP 262.) By the way, under Python 2.5, packages installed by the distutils include an .egg-info *file*, so that is a giveaway that a package was installed using distutils (and maybe a system packaging tool), and is definitely *not* a "develop" package. From cakebread at gmail.com Fri Apr 13 20:51:27 2007 From: cakebread at gmail.com (Rob Cakebread) Date: Fri, 13 Apr 2007 11:51:27 -0700 Subject: [Distutils] Determining system packages In-Reply-To: <5.1.1.6.0.20070413141914.02b139c0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070413141914.02b139c0@sparrow.telecommunity.com> Message-ID: <9b06ffb10704131151x7fdaae26h35257e6f08c18558@mail.gmail.com> On 4/13/07, Phillip J. Eby wrote: > > You mean 'get_python_lib(standard_lib=1)'? > That's much better, thanks. > > System packages are just develop eggs installed in "system" > locations. System locations may be Python-defined or OS-defined. For > example, some distros have non-standard directory layouts, such that there > is both a "/usr/lib/python2.x/site-packages" and a > "/usr/local/lib/python2.x/site-packages", and *either* could contain system > packages. > I think I'll wait till I finish the plugin system for package mangers before I try to guess if its a system package then. If a package manager has a manifest for Python itself I'll be able to determine that wsigref for instance is "owned" by the Python package and therefore a system package. > There is a trick you could use to identify "develop" distributions, > though... if there is a "SOURCES.txt" metadata file associated with an > egg, you can look for a setup.py in a parent directory whose directory > layout corresponds to that described by the contents of SOURCES.txt. This > could *potentially* produce a false positive, though, if the package in > question puts all its modules in the project root and doesn't have any > other files besides setup.py listed in the project root. > > (The long-term plan for 0.7 is that packages that are actually "installed" > will have some kind of installation manifest ala PEP 262.) > > By the way, under Python 2.5, packages installed by the distutils include > an .egg-info *file*, so that is a giveaway that a package was installed > using distutils (and maybe a system packaging tool), and is definitely > *not* a "develop" package. > Ok, great, thanks. I'll use those methods to check also. Rob From michael.dunstan at gmail.com Sat Apr 14 03:33:38 2007 From: michael.dunstan at gmail.com (Michael Dunstan) Date: Sat, 14 Apr 2007 13:33:38 +1200 Subject: [Distutils] easy_install fails with tarfile.ExtractError: could not change mode Message-ID: <99b561f50704131833r3e819adbgdcdda6bc878a0f94@mail.gmail.com> Hi, I've found that for easy_install fails to install zc.recipe.testrunner for python 2.3 on OS X. % easy_install-2.3 zc.recipe.testrunner Searching for zc.recipe.testrunner Reading http://cheeseshop.python.org/pypi/zc.recipe.testrunner/ Reading http://svn.zope.org/zc.buildout Reading http://cheeseshop.python.org/pypi/zc.recipe.testrunner/1.0.0b6 Best match: zc.recipe.testrunner 1.0.0b6 Downloading http://cheeseshop.python.org/packages/source/z/zc.recipe.testrunner/zc.recipe.testrunner-1.0.0b6.tar.gz#md5=108e0e4b961cf6301b49a85678b8f0a9 Processing zc.recipe.testrunner-1.0.0b6.tar.gz Traceback (most recent call last): File "/opt/local/bin/easy_install-2.3", line 7, in ? sys.exit( File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 1670, in main with_ei_usage(lambda: File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 1659, in with_ei_usage return f() File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 1674, in distclass=DistributionWithoutHelpCommands, **kw File "/opt/local/lib/python2.3/distutils/core.py", line 149, in setup dist.run_commands() File "/opt/local/lib/python2.3/distutils/dist.py", line 907, in run_commands self.run_command(cmd) File "/opt/local/lib/python2.3/distutils/dist.py", line 927, in run_command cmd_obj.run() File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 211, in run self.easy_install(spec, not self.no_deps) File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 446, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 471, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/command/easy_install.py", line 626, in install_eggs unpack_archive(dist_filename, tmpdir, self.unpack_progress) File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/archive_util.py", line 67, in unpack_archive driver(filename, extract_dir, progress_filter) File "/opt/local/lib/python2.3/site-packages/setuptools-0.6c5-py2.3.egg/setuptools/archive_util.py", line 192, in unpack_tarfile tarobj._extract_member(member,dst) # XXX Ugh File "/opt/local/lib/python2.3/tarfile.py", line 1414, in _extract_member self.chmod(tarinfo, targetpath) File "/opt/local/lib/python2.3/tarfile.py", line 1532, in chmod raise ExtractError, "could not change mode" tarfile.ExtractError: could not change mode zsh: 1148 exit 1 easy_install-2.3 zc.recipe.testrunner The root of the problem seems to a file mode with the set-group-ID-on-execution bit set in the source distribution of zc.recipe.testrunner. Here is a demonstration of that: >>> import os >>> import tempfile >>> tmp_dir = tempfile.mkdtemp(prefix="easy_install-") >>> example_dir = os.path.join(tmp_dir, 'example') >>> os.mkdir(example_dir) >>> os.chmod(example_dir, 02755) Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 1] Operation not permitted: '/tmp/easy_install-UlgpCH/example' I see the same failing behaviour for handling of tempfile and chmod for Python 2.4 on OS X. However there is an explicit egg for python 2.4: http://cheeseshop.python.org/pypi/zc.recipe.testrunner/1.0.0b6#download which does not have set-group-ID-on-execution bit set for any of the archive members. So the egg easy installs just fine in that case. And I don't see any such problems using FreeBSD. Michael From robinbryce at gmail.com Sat Apr 14 05:53:37 2007 From: robinbryce at gmail.com (Robin Bryce) Date: Sat, 14 Apr 2007 04:53:37 +0100 Subject: [Distutils] [ANN] pypicache 0.1 Message-ID: Hi All, I've just uploaded pypicache 0.1 to the PyPI index. I think it might be interesting to readers of this list. pypicache aims to be a relatively simple way to produce &maintain a personal package index - possibly just your PYTHON_EGG_CACHE directory - and have it published to easy_install over a LAN or intranet. Please let me know what you make of pypicache. I'm especially interested in whether or not I've correctly implemented the Project/Page mappings and down load urls. And in general please point out any glaringly bad ideas ! I'm currently hosting a small example index at http://svn.wiretooth.com/pypi/. There is a small 'first use' guide based on the steps I took to this index up and running: http://svn.wiretooth.com/svn/open/pypicache/trunk/example-firstuse.html Some more links general info: http://svn.wiretooth.com/svn/open/pypicache/tags/release-0.1/pypicache.html tool help: http://svn.wiretooth.com/svn/open/pypicache/tags/release-0.1/indexpackages-man.html The tool produces exactly one html page per (safe) project/version combination and generates a single mod-rewrite text map to provide the /Project/, Project/VER uri name space. Everything it generates, apache configuration and project pages, is contained in the index directory beside the distributions. Robin From jim at zope.com Sat Apr 14 14:47:47 2007 From: jim at zope.com (Jim Fulton) Date: Sat, 14 Apr 2007 08:47:47 -0400 Subject: [Distutils] easy_install fails with tarfile.ExtractError: could not change mode In-Reply-To: <99b561f50704131833r3e819adbgdcdda6bc878a0f94@mail.gmail.com> References: <99b561f50704131833r3e819adbgdcdda6bc878a0f94@mail.gmail.com> Message-ID: On Apr 13, 2007, at 9:33 PM, Michael Dunstan wrote: > I've found that for easy_install fails to install zc.recipe.testrunner > for python 2.3 on OS X. ... > The root of the problem seems to a file mode with the > set-group-ID-on-execution bit set in the source distribution of > zc.recipe.testrunner. Weird. Thanks for chasing this down. I've uploaded a new release without that bit set. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From michael.dunstan at gmail.com Sat Apr 14 23:08:10 2007 From: michael.dunstan at gmail.com (Michael Dunstan) Date: Sun, 15 Apr 2007 09:08:10 +1200 Subject: [Distutils] easy_install fails with tarfile.ExtractError: could not change mode In-Reply-To: References: <99b561f50704131833r3e819adbgdcdda6bc878a0f94@mail.gmail.com> Message-ID: <99b561f50704141408t730e0b1blc07f98893c67fad6@mail.gmail.com> On 4/15/07, Jim Fulton wrote: > Weird. Thanks for chasing this down. I've uploaded a new release > without that bit set. Thanks. That does the trick nicely. Michael From fccoelho at gmail.com Tue Apr 17 21:16:51 2007 From: fccoelho at gmail.com (=?ISO-8859-1?Q?Fl=E1vio_Code=E7o?= Coelho) Date: Tue, 17 Apr 2007 16:16:51 -0300 Subject: [Distutils] sandbox violation Message-ID: <1176837411.16527.21.camel@flavio-desktop> Hi, I have a setup.py that has recently stopped working with a sandbox-violation error message. During installation I install files under /usr/share Is installation of file outside the python path forbidden now? Is ther any workaraound? thanks, Fl?vio -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070417/d59a07d4/attachment.htm From pje at telecommunity.com Wed Apr 18 02:04:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 17 Apr 2007 20:04:30 -0400 Subject: [Distutils] sandbox violation In-Reply-To: <1176837411.16527.21.camel@flavio-desktop> Message-ID: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> At 04:16 PM 4/17/2007 -0300, Fl?vio Code?o Coelho wrote: >Hi, > >I have a setup.py that has recently stopped working with a >sandbox-violation error message. During installation I install files under >/usr/share > >Is installation of file outside the python path forbidden now? It is when an egg is being built, since anything not installed to the temporary directory where the egg is being built will not get included in the egg. >Is ther any workaraound? Don't use easy_install or build an egg. Eggs must be self-contained, and easy_install builds eggs before installing them. That's a feature of eggs -- i.e., they don't put files all over your system. From fccoelho at fiocruz.br Wed Apr 18 13:52:18 2007 From: fccoelho at fiocruz.br (=?ISO-8859-1?Q?Fl=E1vio_Code=E7o?= Coelho) Date: Wed, 18 Apr 2007 08:52:18 -0300 Subject: [Distutils] sandbox violation In-Reply-To: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> References: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> Message-ID: <1176897138.16527.47.camel@flavio-desktop> Em Ter, 2007-04-17 ?s 20:04 -0400, Phillip J. Eby escreveu: > At 04:16 PM 4/17/2007 -0300, Fl?vio Code?o Coelho wrote: > >Hi, > > > >I have a setup.py that has recently stopped working with a > >sandbox-violation error message. During installation I install files under > >/usr/share > > > >Is installation of file outside the python path forbidden now? > > It is when an egg is being built, since anything not installed to the > temporary directory where the egg is being built will not get included in > the egg. > Is this a recent change to easy_install? cause my packages used to install nicely with easy_instal up to a few weeks ago... > > >Is ther any workaraound? > > Don't use easy_install or build an egg. Eggs must be self-contained, and > easy_install builds eggs before installing them. That's a feature of eggs > -- i.e., they don't put files all over your system. I agree with this philosophy. But does it mean that if want my packages to install files to other locations, I am forever banned from the setuptools world? ;-) Please say it is not true...;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070418/2875ed91/attachment.html From pje at telecommunity.com Wed Apr 18 17:47:14 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 18 Apr 2007 11:47:14 -0400 Subject: [Distutils] sandbox violation In-Reply-To: <1176897138.16527.47.camel@flavio-desktop> References: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070418113939.04d19008@sparrow.telecommunity.com> At 08:52 AM 4/18/2007 -0300, Fl?vio Code?o Coelho wrote: >Em Ter, 2007-04-17 ?s 20:04 -0400, Phillip J. Eby escreveu: >> >> >>At 04:16 PM 4/17/2007 -0300, Fl?vio Code?o Coelho wrote: >> >Hi, >> > >> >I have a setup.py that has recently stopped working with a >> >sandbox-violation error message. During installation I install files under >> >/usr/share >> > >> >Is installation of file outside the python path forbidden now? >> >>It is when an egg is being built, since anything not installed to the >>temporary directory where the egg is being built will not get included in >>the egg. >> >Is this a recent change to easy_install? No; it was originally implemented a little less than 2 years ago: http://svn.python.org/view?rev=41029&view=rev > cause my packages used to install nicely with easy_instal up to a few > weeks ago... Evidently you've changed something else, or you've been using a 2-year-old version of easy_install. :) >> >Is ther any workaraound? >> >>Don't use easy_install or build an egg. Eggs must be self-contained, and >>easy_install builds eggs before installing them. That's a feature of eggs >>-- i.e., they don't put files all over your system. > >I agree with this philosophy. But does it mean that if want my packages to >install files to other locations, I am forever banned from the >setuptools world? ;-) Please say it is not true...;-) What you must not do is install files outside the --install-dir of the distutils install_data or install_lib commands, which is rude even in the distutils-only world. In other words, do not use absolute paths in your setup() data_files. If you need to have a different default --install-dir for these commands, you should put that information in your project's setup.cfg. When building an egg, your defaults will then be ignored (because bdist_egg explicitly sets its own directories), but when your project is installed in backward-compatibility mode (i.e., not as a .egg file but as a direct install or system package), the data will go where you want it. From tseaver at palladion.com Wed Apr 18 17:46:09 2007 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 18 Apr 2007 11:46:09 -0400 Subject: [Distutils] sandbox violation In-Reply-To: <1176897138.16527.47.camel@flavio-desktop> References: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> <1176897138.16527.47.camel@flavio-desktop> Message-ID: <46263D41.3020703@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Fl?vio Code?o Coelho wrote: > Em Ter, 2007-04-17 ?s 20:04 -0400, Phillip J. Eby escreveu: > >> At 04:16 PM 4/17/2007 -0300, Fl?vio Code?o Coelho wrote: >>> Hi, >>> >>> I have a setup.py that has recently stopped working with a >>> sandbox-violation error message. During installation I install files under >>> /usr/share >>> >>> Is installation of file outside the python path forbidden now? >> It is when an egg is being built, since anything not installed to the >> temporary directory where the egg is being built will not get included in >> the egg. >> > > Is this a recent change to easy_install? cause my packages used to > install nicely with easy_instal up to a few weeks ago... > >>> Is ther any workaraound? >> Don't use easy_install or build an egg. Eggs must be self-contained, and >> easy_install builds eggs before installing them. That's a feature of eggs >> -- i.e., they don't put files all over your system. > > > I agree with this philosophy. But does it mean that if want my packages > to install files to other locations, I am forever banned from the > setuptools world? ;-) Please say it is not true...;-) I think I understood Phillip's response to say that you can *install* files outside of the egg; you just can't put them there during the *build* phase, which is when the sandboxing is enforced. This is "best practice" for other systems as well: people would be extremely upset if 'make' by itself deployed files outside the local directory, rather than doing that effort during 'make install'. An additional benefit is that users can *build* the egg without any special privileges (as with RPMs, for instance). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGJj1B+gerLs4ltQ4RAjgiAJ9AqtOx1paeSU/dNefRUpP/ky6UnACdEE0R nUjsqu4s/DpZcdJ8vifULqs= =sGK6 -----END PGP SIGNATURE----- From tseaver at palladion.com Wed Apr 18 17:46:09 2007 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 18 Apr 2007 11:46:09 -0400 Subject: [Distutils] sandbox violation In-Reply-To: <1176897138.16527.47.camel@flavio-desktop> References: <5.1.1.6.0.20070417200158.0297f598@sparrow.telecommunity.com> <1176897138.16527.47.camel@flavio-desktop> Message-ID: <46263D41.3020703@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Fl?vio Code?o Coelho wrote: > Em Ter, 2007-04-17 ?s 20:04 -0400, Phillip J. Eby escreveu: > >> At 04:16 PM 4/17/2007 -0300, Fl?vio Code?o Coelho wrote: >>> Hi, >>> >>> I have a setup.py that has recently stopped working with a >>> sandbox-violation error message. During installation I install files under >>> /usr/share >>> >>> Is installation of file outside the python path forbidden now? >> It is when an egg is being built, since anything not installed to the >> temporary directory where the egg is being built will not get included in >> the egg. >> > > Is this a recent change to easy_install? cause my packages used to > install nicely with easy_instal up to a few weeks ago... > >>> Is ther any workaraound? >> Don't use easy_install or build an egg. Eggs must be self-contained, and >> easy_install builds eggs before installing them. That's a feature of eggs >> -- i.e., they don't put files all over your system. > > > I agree with this philosophy. But does it mean that if want my packages > to install files to other locations, I am forever banned from the > setuptools world? ;-) Please say it is not true...;-) I think I understood Phillip's response to say that you can *install* files outside of the egg; you just can't put them there during the *build* phase, which is when the sandboxing is enforced. This is "best practice" for other systems as well: people would be extremely upset if 'make' by itself deployed files outside the local directory, rather than doing that effort during 'make install'. An additional benefit is that users can *build* the egg without any special privileges (as with RPMs, for instance). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGJj1B+gerLs4ltQ4RAjgiAJ9AqtOx1paeSU/dNefRUpP/ky6UnACdEE0R nUjsqu4s/DpZcdJ8vifULqs= =sGK6 -----END PGP SIGNATURE----- From jkp at kirkconsulting.co.uk Fri Apr 20 13:26:06 2007 From: jkp at kirkconsulting.co.uk (Jamie Kirkpatrick) Date: Fri, 20 Apr 2007 12:26:06 +0100 Subject: [Distutils] BUG: Stub loaders not created for non distutils .pyd files when using setuptools. Message-ID: All I have come across what I believe is a bug in setuputils bdist_egg command. According to this page http://peak.telecommunity.com/ DevCenter/EggFormats#zip-file-issues it is not possible for python's zip loader to load shared libraries directly from the zip files themselves therefore a special stub loader py file is generated for each C extension included in a package that works around this issue. The problem I have found becomes apparent when pyd extension libraries are compiled using a process that lives outside of distutils itself (ie not using the Extension class). The code in bdist_egg.py only generates stub loaders for extensions that are built by distutils - any pyd files included in another manner are not processed. This is actually inconsistent with a method elsewhere in the distribution called exe_to_egg or somesuch where all resources are scanned and any .pyd files get a stub generated. I've created a simple patch (attached) to fix the issue. It checks all the resources found in the all_outputs list and generates stubs for any pyd files found - this covers off cases where you have pyd files included that are not generated by distutils itself. I understand that the preferred method would be to use distutils to do all extension builds but in my case this is not possible or desirable. I'm dynamically generating a set of extensions as the result of a toolchain that includes SWIG and needs to be refused across multiple languages so we use Ribosome to build the actual extension files in a portable manner instead of distutils. I then copy them into the final package as datafiles. I hope that either my patch (or a modified version) can be incorporated or someone can suggest another workaround that would fix the issue allowing externally generated pyd files to have wrappers auto-generated by bdist_egg. Many thanks in advance JKP -------------- next part -------------- A non-text attachment was scrubbed... Name: bdist_egg+stub_creation.patch Type: application/octet-stream Size: 1003 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070420/5978c5e4/attachment.obj From pje at telecommunity.com Fri Apr 20 16:26:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Apr 2007 10:26:53 -0400 Subject: [Distutils] BUG: Stub loaders not created for non distutils .pyd files when using setuptools. In-Reply-To: Message-ID: <5.1.1.6.0.20070420102324.04f51ea8@sparrow.telecommunity.com> At 12:26 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote: >I hope that either my patch (or a modified version) can be >incorporated or someone can suggest another workaround that would fix >the issue allowing externally generated pyd files to have wrappers >auto-generated by bdist_egg. Have you tried something like this? from setuptools.command.build_ext import build_ext class my_build_ext(build_ext): def run(self): pass setup( cmdclass = {'build_ext':my_build_ext}, ext_modules = [Extension(...), ....] } (That is, simply using a custom build_ext subclass that doesn't do anything... or perhaps invokes your custom build process.) From jkp at kirkconsulting.co.uk Fri Apr 20 17:15:13 2007 From: jkp at kirkconsulting.co.uk (Jamie Kirkpatrick) Date: Fri, 20 Apr 2007 16:15:13 +0100 Subject: [Distutils] BUG: Stub loaders not created for non distutils .pyd files when using setuptools. In-Reply-To: <5.1.1.6.0.20070420102324.04f51ea8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070420102324.04f51ea8@sparrow.telecommunity.com> Message-ID: I haven't tried this - looks like it would be worth a go for sure. If so then I guess it can be considered a workaround for the "issue". Question though - how will the setuptools machinery know what the .pyd file is called? Jamie On 20 Apr 2007, at 15:26, Phillip J. Eby wrote: > At 12:26 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote: >> I hope that either my patch (or a modified version) can be >> incorporated or someone can suggest another workaround that would fix >> the issue allowing externally generated pyd files to have wrappers >> auto-generated by bdist_egg. > > Have you tried something like this? > > from setuptools.command.build_ext import build_ext > class my_build_ext(build_ext): > def run(self): pass > > setup( > cmdclass = {'build_ext':my_build_ext}, > ext_modules = [Extension(...), ....] > } > > (That is, simply using a custom build_ext subclass that doesn't do > anything... or perhaps invokes your custom build process.) > From pje at telecommunity.com Fri Apr 20 17:58:09 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Apr 2007 11:58:09 -0400 Subject: [Distutils] BUG: Stub loaders not created for non distutils .pyd files when using setuptools. In-Reply-To: References: <5.1.1.6.0.20070420102324.04f51ea8@sparrow.telecommunity.com> <5.1.1.6.0.20070420102324.04f51ea8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070420115612.02b49218@sparrow.telecommunity.com> At 04:15 PM 4/20/2007 +0100, Jamie Kirkpatrick wrote: >I haven't tried this - looks like it would be worth a go for sure. >If so then I guess it can be considered a workaround for the >"issue". Question though - how will the setuptools machinery know >what the .pyd file is called? By you giving the correct information to Extension() as if you were going to have the distutils do the building. From ianb at colorstudy.com Sat Apr 21 01:11:57 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 20 Apr 2007 18:11:57 -0500 Subject: [Distutils] Derivative files Message-ID: <462948BD.4020500@colorstudy.com> I made a distutils command to take the Javascript files in an application and put them together and compress them. I'd like the result to kind of always be there; it would be distributed with the egg for instance. But I don't really want it in source control. So I think I need to make this part of the build somehow. What should I look for to do that? -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From pje at telecommunity.com Sat Apr 21 01:40:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 20 Apr 2007 19:40:10 -0400 Subject: [Distutils] Derivative files In-Reply-To: <462948BD.4020500@colorstudy.com> Message-ID: <5.1.1.6.0.20070420193832.02fb4ee0@sparrow.telecommunity.com> At 06:11 PM 4/20/2007 -0500, Ian Bicking wrote: >I made a distutils command to take the Javascript files in an >application and put them together and compress them. I'd like the >result to kind of always be there; it would be distributed with the egg >for instance. But I don't really want it in source control. So I think >I need to make this part of the build somehow. What should I look for >to do that? A couple of options: * Add it to MANIFEST.in, in which case it will be put in source distributions as well as the eggs, assuming it's in a package subdirectory and include_package_data is True * list it explicitly in package_data From ianb at colorstudy.com Mon Apr 23 05:22:42 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 22 Apr 2007 22:22:42 -0500 Subject: [Distutils] Derivative files In-Reply-To: <5.1.1.6.0.20070420193832.02fb4ee0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070420193832.02fb4ee0@sparrow.telecommunity.com> Message-ID: <462C2682.5030902@colorstudy.com> Phillip J. Eby wrote: > At 06:11 PM 4/20/2007 -0500, Ian Bicking wrote: >> I made a distutils command to take the Javascript files in an >> application and put them together and compress them. I'd like the >> result to kind of always be there; it would be distributed with the egg >> for instance. But I don't really want it in source control. So I think >> I need to make this part of the build somehow. What should I look for >> to do that? > > A couple of options: > > * Add it to MANIFEST.in, in which case it will be put in source > distributions as well as the eggs, assuming it's in a package > subdirectory and include_package_data is True > > * list it explicitly in package_data Is there a good way to ensure the files are created if necessary? I.e., if someone checks out the code (without these files) and runs setup.py install or develop, to make sure the compressing command is run? -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From pje at telecommunity.com Mon Apr 23 05:32:26 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 22 Apr 2007 23:32:26 -0400 Subject: [Distutils] Derivative files In-Reply-To: <462C2682.5030902@colorstudy.com> References: <5.1.1.6.0.20070420193832.02fb4ee0@sparrow.telecommunity.com> <5.1.1.6.0.20070420193832.02fb4ee0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070422233133.02eaa008@sparrow.telecommunity.com> At 10:22 PM 4/22/2007 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 06:11 PM 4/20/2007 -0500, Ian Bicking wrote: >>>I made a distutils command to take the Javascript files in an >>>application and put them together and compress them. I'd like the >>>result to kind of always be there; it would be distributed with the egg >>>for instance. But I don't really want it in source control. So I think >>>I need to make this part of the build somehow. What should I look for >>>to do that? >>A couple of options: >>* Add it to MANIFEST.in, in which case it will be put in source >>distributions as well as the eggs, assuming it's in a package >>subdirectory and include_package_data is True >>* list it explicitly in package_data > >Is there a good way to ensure the files are created if necessary? I.e., >if someone checks out the code (without these files) and runs setup.py >install or develop, to make sure the compressing command is run? You can subclass one of the build_* or install_* commands that would ordinarily be run. From gentoodev at gmail.com Mon Apr 23 09:04:23 2007 From: gentoodev at gmail.com (Rob Cakebread) Date: Mon, 23 Apr 2007 00:04:23 -0700 Subject: [Distutils] setuptools http auth Message-ID: <9b06ffb10704230004r34f75302nbce8dd7cbaf445fa@mail.gmail.com> Hi, I have http://anonymous at tools.assembla.com/svn/yolk/trunk/#egg=yolk-dev in my PKG-INFO. When I try to use 'easy_install yolk==dev' I get: Best match: yolk dev Downloading http://anonymous@tools.assembla.com/svn/yolk/trunk/#egg=yolk-dev error: Download error for http://anonymous&: (-2, 'Name or service not known') I can see that when I use 'setup.py register' the @ is changed to @ Is there any way around this? Thanks, Rob From pje at telecommunity.com Mon Apr 23 18:12:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Apr 2007 12:12:44 -0400 Subject: [Distutils] setuptools http auth In-Reply-To: <9b06ffb10704230004r34f75302nbce8dd7cbaf445fa@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070423121155.04f6a3a0@sparrow.telecommunity.com> At 12:04 AM 4/23/2007 -0700, Rob Cakebread wrote: >Hi, > >I have http://anonymous at tools.assembla.com/svn/yolk/trunk/#egg=yolk-dev >in my PKG-INFO. > >When I try to use 'easy_install yolk==dev' I get: > >Best match: yolk dev >Downloading >http://anonymous@tools.assembla.com/svn/yolk/trunk/#egg=yolk-dev >error: Download error for http://anonymous&: (-2, 'Name or service not known') > >I can see that when I use 'setup.py register' the @ is changed to @ >Is there any way around this? Argh. setuptools.package_index isn't HTML-unquoting URLs. That needs to be fixed. :( From cakebread at gmail.com Mon Apr 23 22:07:46 2007 From: cakebread at gmail.com (Rob Cakebread) Date: Mon, 23 Apr 2007 13:07:46 -0700 Subject: [Distutils] setuptools http auth In-Reply-To: <5.1.1.6.0.20070423121155.04f6a3a0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070423121155.04f6a3a0@sparrow.telecommunity.com> Message-ID: <9b06ffb10704231307g1ea432e4n767bf3d959d752b2@mail.gmail.com> On 4/23/07, Phillip J. Eby wrote: > > Argh. setuptools.package_index isn't HTML-unquoting URLs. That needs to > be fixed. :( > > I'm not sure if this is the best way to go about it, I couldn't find anything in the stdlib to unencode it, but this works: --- setuptools/package_index.py (revision 54927) +++ setuptools/package_index.py (working copy) @@ -385,6 +385,7 @@ of `tmpdir`, and the local filename is returned. Various errors may be raised if a problem occurs during downloading. """ + spec = spec.replace("@", "@") if not isinstance(spec,Requirement): scheme = URL_SCHEME(spec) if scheme: From pje at telecommunity.com Mon Apr 23 23:55:07 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 23 Apr 2007 17:55:07 -0400 Subject: [Distutils] setuptools http auth In-Reply-To: <9b06ffb10704231307g1ea432e4n767bf3d959d752b2@mail.gmail.co m> References: <5.1.1.6.0.20070423121155.04f6a3a0@sparrow.telecommunity.com> <5.1.1.6.0.20070423121155.04f6a3a0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070423174346.04bf3e00@sparrow.telecommunity.com> At 01:07 PM 4/23/2007 -0700, Rob Cakebread wrote: >On 4/23/07, Phillip J. Eby wrote: >> >>Argh. setuptools.package_index isn't HTML-unquoting URLs. That needs to >>be fixed. :( >> > >I'm not sure if this is the best way to go about it, I couldn't find >anything in the stdlib >to unencode it, but this works: > >--- setuptools/package_index.py (revision 54927) >+++ setuptools/package_index.py (working copy) >@@ -385,6 +385,7 @@ > of `tmpdir`, and the local filename is returned. Various errors > may be > raised if a problem occurs during downloading. > """ >+ spec = spec.replace("@", "@") > if not isinstance(spec,Requirement): > scheme = URL_SCHEME(spec) > if scheme: There are actually a bunch of places where these URLs get used, so the decoding actually needs to happen all over. So I'm implementing a more complete solution. From antoine.pitrou at wengo.com Tue Apr 24 17:27:12 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Tue, 24 Apr 2007 17:27:12 +0200 Subject: [Distutils] programmatically adding entry points Message-ID: <1177428432.27903.3.camel@antoine-ubuntu> hello, First, I don't know if this is the right mailing-list for asking questions about pkg_resources. If it isn't, please be kind to point me to the right place. I've read the pkg_resources API (*) and I don't see any way of programmatically registering a (possibly temporary) entry-point. The only documented way to add entry points is to write setup scripts. But if I want to write unit tests for entry point-related behaviour, I don't want to run setup scripts in my unit tests. Is there a simple way to do this? (*) http://peak.telecommunity.com/DevCenter/PkgResources Thanks Antoine. From pje at telecommunity.com Tue Apr 24 18:45:43 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 24 Apr 2007 12:45:43 -0400 Subject: [Distutils] programmatically adding entry points In-Reply-To: <1177428432.27903.3.camel@antoine-ubuntu> Message-ID: <5.1.1.6.0.20070424123900.03d97818@sparrow.telecommunity.com> At 05:27 PM 4/24/2007 +0200, Antoine Pitrou wrote: >I've read the pkg_resources API (*) and I don't see any way of >programmatically registering a (possibly temporary) entry-point. The >only documented way to add entry points is to write setup scripts. >But if I want to write unit tests for entry point-related behaviour, I >don't want to run setup scripts in my unit tests. Is there a simple way >to do this? Write your code to separate the finding of entry points, from the *processing* of entry points. Then pass in manually-created EntryPoint instances to test the processing routines. Alternately, pass a mock WorkingSet to your routines that find entry points. That is, write them like this: def some_routine(working_set=pkg_resources.working_set): for blah in working_set.iter_entry_points('whatever'): ... And then just pass in a mock object for testing, that yields the EntryPoint instances you want to test. From antoine.pitrou at wengo.com Tue Apr 24 18:50:15 2007 From: antoine.pitrou at wengo.com (Antoine Pitrou) Date: Tue, 24 Apr 2007 18:50:15 +0200 Subject: [Distutils] programmatically adding entry points In-Reply-To: <5.1.1.6.0.20070424123900.03d97818@sparrow.telecommunity.com> References: <5.1.1.6.0.20070424123900.03d97818@sparrow.telecommunity.com> Message-ID: <1177433415.27903.4.camel@antoine-ubuntu> Le mardi 24 avril 2007 ? 12:45 -0400, Phillip J. Eby a ?crit : > At 05:27 PM 4/24/2007 +0200, Antoine Pitrou wrote: > >I've read the pkg_resources API (*) and I don't see any way of > >programmatically registering a (possibly temporary) entry-point. The > >only documented way to add entry points is to write setup scripts. > >But if I want to write unit tests for entry point-related behaviour, I > >don't want to run setup scripts in my unit tests. Is there a simple way > >to do this? > > Write your code to separate the finding of entry points, from the > *processing* of entry points. Then pass in manually-created EntryPoint > instances to test the processing routines. Well, it makes sense. Thanks :-) Antoine. From sklein at cpcug.org Tue Apr 24 22:26:21 2007 From: sklein at cpcug.org (Stanley A. Klein) Date: Tue, 24 Apr 2007 16:26:21 -0400 (EDT) Subject: [Distutils] I need to create a customized Fedora live cd with Python applications on it Message-ID: <64958.207.188.248.157.1177446381.squirrel@www.cpcug.org> I need to create a customized Fedora live CD with Python applications on it. The live CD requires rpm's of the applications. (The issues are similar to a customized Ubuntu live CD but Ubuntu requires deb's.) It seems that the distutils/setup.py functions that create rpms are broken. I get errors most of the time that I try to use them. The Python community seems to be focused on eggs, that are useful if individually installed but don't match the distribution formats of the Linux distros. Those distribution formats are what the distros use for their live CD's. Is there a way to convert eggs to rpm's? Can the distutils be fixed to reliably create rpm's? Is there some other way to install eggs on distro customized live CD's? Stan Klein From nathan at yergler.net Tue Apr 24 22:50:51 2007 From: nathan at yergler.net (Nathan R. Yergler) Date: Tue, 24 Apr 2007 16:50:51 -0400 Subject: [Distutils] I need to create a customized Fedora live cd with Python applications on it In-Reply-To: <64958.207.188.248.157.1177446381.squirrel@www.cpcug.org> References: <64958.207.188.248.157.1177446381.squirrel@www.cpcug.org> Message-ID: On 4/24/07, Stanley A. Klein wrote: > I need to create a customized Fedora live CD with Python applications on > it. The live CD requires rpm's of the applications. (The issues are > similar to a customized Ubuntu live CD but Ubuntu requires deb's.) > > It seems that the distutils/setup.py functions that create rpms are > broken. I get errors most of the time that I try to use them. The Python > community seems to be focused on eggs, that are useful if individually > installed but don't match the distribution formats of the Linux distros. > Those distribution formats are what the distros use for their live CD's. I've successfully created RPMs many times in the past; can you elaborate on the problems you're having? > > Is there a way to convert eggs to rpm's? Can the distutils be fixed to > reliably create rpm's? Is there some other way to install eggs on distro > customized live CD's? If it's a live CD, is there a reason you can't just put them into site-packages on the CD filesystem? (I have no idea what goes into building a Fedora live CD, so "yes, there's a reason" is a sufficient answer). > > > Stan Klein > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > From kumar.mcmillan at gmail.com Thu Apr 26 19:50:36 2007 From: kumar.mcmillan at gmail.com (Kumar McMillan) Date: Thu, 26 Apr 2007 12:50:36 -0500 Subject: [Distutils] setuptools extras_require In-Reply-To: <5.1.1.6.0.20070209154357.0500fe68@sparrow.telecommunity.com> References: <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> <1166082725.27648.50.camel@nny> <45CC853F.5020908@palladion.com> <5.1.1.6.0.20070209154357.0500fe68@sparrow.telecommunity.com> Message-ID: On 2/9/07, Phillip J. Eby wrote: > At 09:29 AM 2/9/2007 -0500, Tres Seaver wrote: > > > Using "|" as a separator for ORing TurboGear's requirements might look > > > something like this (ignoring version numbers for simplicity): > > > > > > install_requires = [ > > > "SQLObject | SQLAlchemy", > > > "Kid | Genshi", > > > ], > > > extras_require = { > > > "fancy_tb": ["Paste"], > > > } > > > >+1 for the pipe symbol: to my eye, it means "pick the first of these", > >which is what you want. > > It's also implementable; the tuple concept isn't. Eggs' requirements get > stored in text files, so there has to be a pure-text representation possible. [blast from the past] I'd like to use ORed requirements in a setup file but I don't see what version of setuptools this was introduced in. Does anyone know? I was looking at: http://peak.telecommunity.com/DevCenter/EasyInstall#release-notes-change-history thanks, Kumar From pje at telecommunity.com Thu Apr 26 22:07:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 26 Apr 2007 16:07:00 -0400 Subject: [Distutils] setuptools extras_require In-Reply-To: References: <5.1.1.6.0.20070209154357.0500fe68@sparrow.telecommunity.com> <5.1.1.6.0.20061210155520.029828f0@sparrow.telecommunity.com> <5035593a0612111344w7f5c3490t9c0424aa863fabcc@mail.gmail.com> <5035593a0612120732j7eaec5anb48234c2b056d9b0@mail.gmail.com> <1165941302.6958.23.camel@nny> <5035593a0612132215u3c08dda0h7f2f6fd9011c1697@mail.gmail.com> <1166082725.27648.50.camel@nny> <45CC853F.5020908@palladion.com> <5.1.1.6.0.20070209154357.0500fe68@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070426160408.02e06f28@sparrow.telecommunity.com> At 12:50 PM 4/26/2007 -0500, Kumar McMillan wrote: >On 2/9/07, Phillip J. Eby wrote: >>At 09:29 AM 2/9/2007 -0500, Tres Seaver wrote: >> > > Using "|" as a separator for ORing TurboGear's requirements might look >> > > something like this (ignoring version numbers for simplicity): >> > > >> > > install_requires = [ >> > > "SQLObject | SQLAlchemy", >> > > "Kid | Genshi", >> > > ], >> > > extras_require = { >> > > "fancy_tb": ["Paste"], >> > > } >> > >> >+1 for the pipe symbol: to my eye, it means "pick the first of these", >> >which is what you want. >> >>It's also implementable; the tuple concept isn't. Eggs' requirements get >>stored in text files, so there has to be a pure-text representation possible. > >[blast from the past] >I'd like to use ORed requirements in a setup file but I don't see what >version of setuptools this was introduced in. Does anyone know? No, because it hasn't been done yet. :) I merely said it was implementable (in principle). Note that any project using this format won't be compatible with 0.6 versions of setuptools and will cause them to fail installation with parsing errors, unless there is some way to keep the alternatives hidden from older parsers. From optilude at gmx.net Sat Apr 28 18:05:37 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sat, 28 Apr 2007 17:05:37 +0100 Subject: [Distutils] Declaring "already provided" modules Message-ID: Hi guys, I have a particular problem which I think we'll see a lot more of going forward. Zope 3 is in the process of being eggified, and we have eggs in the cheesehop of things like zope.component 3.4.0a1. I use zc.buildout to configure a Zope instance (a zope 2.10 one, to be precise) which has the various zope.* packages in non-egg-format in $INSTANCE_HOME/lib/python/zope/*. Then I'm depending on a package which does this: install_requires=['setuptools', ''zope.component', 'zope.interface', 'zope.schema', ], The problem is that even though I know I have zope.component 3.3 from my Zope instance, setuptools/buildout doesn't know that and tries to resolve the dependency. It succeeds with Zope 3.4's zope.component, which then causes runtime conflicts and subtle problems all over the place. I think what I need is some way of saying "all of these dependencies are already resolved, don't try to get them as new eggs". Is that possible? Is there another way I can manage this? Martin From ianb at colorstudy.com Mon Apr 30 05:41:40 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 29 Apr 2007 22:41:40 -0500 Subject: [Distutils] workingenv/setuptools compatibility In-Reply-To: <5.1.1.6.0.20070220173401.0295e790@sparrow.telecommunity.com> References: <5.1.1.6.0.20070220173401.0295e790@sparrow.telecommunity.com> Message-ID: <46356574.2040006@colorstudy.com> Phillip J. Eby wrote: > At 04:04 PM 2/20/2007 -0600, Ian Bicking wrote: >> Hi again. I mentioned this a long time ago, but I'd like to talk about >> it again: I'd really like to remove workingenv's monkeypatching of >> setuptools. > > Then remove it. :) > > You can trivially replace any setuptools built-in commands by simply > ensuring that you have corresponding 'distutils.commands' entry points > in the working set ahead of setuptools. An easy way to do this in > workingenv would be to place your own egg in the easy-install.pth, and > removing setuptools' egg from there (leaving it in setuptools.pth). > > This will then let you override the 'easy_install' and 'develop' > commands, to do whatever you want them to. I was getting ready to start this, and then I realized it won't actually fix the problem I'm having. workingenv works fine until, for some reason, easy-install.pth changes, setuptools is reinstalled, or something along those lines happens. Then my monkeypatch breaks. I fear that relying on my package to be above setuptools in easy-install.pth might be just as fragile. Though perhaps if I put it in early-install.pth or something like that it would stay above setuptools? > The only pieces this won't affect are the way setup_requires and > tests_require get handled (as they use direct imports from > setuptools.commands.easy_install) and the way > --single-version-externally-managed installs get their scripts generated > (which also uses direct imports). > > However, if I understand correctly, you don't need either of these pieces. > > Anyway, to do all this, you will need to have an egg that lists entry > points for 'easy_install' and 'develop' under the 'distutils.commands' > group. > > (Actually, you might want to include an entry point for 'install' as > well, and override the do_egg_install method, which currently imports > easy_install directly instead of using > self.distribution.get_command_class('easy_install'). I'll fix that in > 0.6c6.) Did these fixes get in, or should I look for other things I'd have to replace/subclass? -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org | Write code, do good | http://topp.openplans.org/careers From pje at telecommunity.com Mon Apr 30 17:24:47 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 30 Apr 2007 11:24:47 -0400 Subject: [Distutils] workingenv/setuptools compatibility In-Reply-To: <46356574.2040006@colorstudy.com> References: <5.1.1.6.0.20070220173401.0295e790@sparrow.telecommunity.com> <5.1.1.6.0.20070220173401.0295e790@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070430112024.02e5e808@sparrow.telecommunity.com> At 10:41 PM 4/29/2007 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>At 04:04 PM 2/20/2007 -0600, Ian Bicking wrote: >>>Hi again. I mentioned this a long time ago, but I'd like to talk about >>>it again: I'd really like to remove workingenv's monkeypatching of >>>setuptools. >>Then remove it. :) >>You can trivially replace any setuptools built-in commands by simply >>ensuring that you have corresponding 'distutils.commands' entry points in >>the working set ahead of setuptools. An easy way to do this in >>workingenv would be to place your own egg in the easy-install.pth, and >>removing setuptools' egg from there (leaving it in setuptools.pth). >>This will then let you override the 'easy_install' and 'develop' >>commands, to do whatever you want them to. > >I was getting ready to start this, and then I realized it won't actually >fix the problem I'm having. workingenv works fine until, for some reason, >easy-install.pth changes, setuptools is reinstalled, or something along >those lines happens. Then my monkeypatch breaks. I fear that relying on >my package to be above setuptools in easy-install.pth might be just as >fragile. Though perhaps if I put it in early-install.pth or something >like that it would stay above setuptools? As long as you install it the way eggs get installed (i.e., with the Python code in the .pth file to push it to the head and track it), yes. >>The only pieces this won't affect are the way setup_requires and >>tests_require get handled (as they use direct imports from >>setuptools.commands.easy_install) and the way >>--single-version-externally-managed installs get their scripts generated >>(which also uses direct imports). >>However, if I understand correctly, you don't need either of these pieces. >>Anyway, to do all this, you will need to have an egg that lists entry >>points for 'easy_install' and 'develop' under the 'distutils.commands' group. >>(Actually, you might want to include an entry point for 'install' as >>well, and override the do_egg_install method, which currently imports >>easy_install directly instead of using >>self.distribution.get_command_class('easy_install'). I'll fix that in 0.6c6.) > >Did these fixes get in, or should I look for other things I'd have to >replace/subclass? No, they haven't gone in yet. Anything that directly imports easy_install (or anything else you're overriding) needs to be changed to use the distribution object's get_command_class('easy_install') instead.