From ronaldoussoren at mac.com Tue Jan 2 16:24:43 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 2 Jan 2007 16:24:43 +0100 Subject: [Distutils] [setuptools] egg_info and install_requires Message-ID: <23480D85-18E8-4456-87C4-0D7A12C7583B@mac.com> Hi, I have a setup.py that has some projects in the install_requires list that setuptools cannot find yet (I haven't build the eggs yet) and noticed something odd. When there is not yet an .egg-info directory for my project everything is fine, I can run distutils commands (except install of course). However, when the .egg-info directory is present all distutils commands fail, including the egg_info command, because I'm refering to a non-existing distribution. Is this supposed to happen? Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070102/91f1bbe5/attachment.bin From schwehr at gmail.com Wed Jan 3 16:47:17 2007 From: schwehr at gmail.com (Kurt Schwehr) Date: Wed, 3 Jan 2007 10:47:17 -0500 Subject: [Distutils] Skipping namespace file? Message-ID: <37de72610701030747k60d074d3o1d034a2a9db23416@mail.gmail.com> Hi All, I am working on packaging pydap and am getting some strange behavior. Here is the install phase... /sw/bin/python2.4 setup.py install --root=/sw/src/fink.build/root- dap-py24-2.2.5.7-1 --single-version-externally-managed And when it gets to the __init__.py file for dap, it skips it. This causes the package to not work. Anyone have an idea why it is doing this? copying dap/wsgi/paster_templates/template/index.tmpl -> build/lib/dap/wsgi/paster_templates/template running install_lib Skipping installation of /sw/src/fink.build/root-dap-py24-2.2.5.7-1/sw/lib/python2.4/site-packages/dap/__init__.py (namespace package) copying dap/client.py -> /sw/src/fink.build/root-dap-py24-2.2.5.7-1 /sw/lib/python2.4/site-packages/dap I am using this version of pydap: http://cheeseshop.python.org/packages/source/d/dap/dap-2.2.5.7.tar.gz This install proceedure works well with other packages in mac osx/fink. I also see the behavior without the --single-version-externally-managed flag. Anyone have a suggestion as to what's going on? Thanks!! -kurt http://schwehr.org/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070103/262a59e2/attachment.htm From jim at zope.com Thu Jan 4 22:14:23 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 4 Jan 2007 16:14:23 -0500 Subject: [Distutils] tests_require and test_suite in egg-info Message-ID: It appears that the values of the tests_require and test_suite keyword arguments isn't captured in egg-info. Am I missing something? Why would this information be omitted? I would like to use it when generating test runners in zc.buildout. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Thu Jan 4 22:37:22 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 04 Jan 2007 16:37:22 -0500 Subject: [Distutils] tests_require and test_suite in egg-info In-Reply-To: Message-ID: <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> At 04:14 PM 1/4/2007 -0500, Jim Fulton wrote: >It appears that the values of the tests_require and test_suite >keyword arguments isn't captured in egg-info. Am I missing >something? Why would this information be omitted? Because the only existing tool that uses that information (the test command) can't be run without the setup.py, which isn't installed with the egg. In any case, many projects treat tests as development tools only, rather than installing them with the project's main code. > I would like to >use it when generating test runners in zc.buildout. Well, you could add an egg_info plugin writer to do it, or you could add a custom setuptools command like "create_test_runners" and run setup.py on it. There are probably also other ways I haven't thought of. :) From pje at telecommunity.com Thu Jan 4 22:42:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 04 Jan 2007 16:42:59 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <37de72610701030747k60d074d3o1d034a2a9db23416@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> At 10:47 AM 1/3/2007 -0500, Kurt Schwehr wrote: >Hi All, > >I am working on packaging pydap and am getting some strange behavior. > >Here is the install phase... > >/sw/bin/python2.4 setup.py install >--root=/sw/src/fink.build/root-dap-py24-2.2.5.7-1 >--single-version-externally-managed > > >And when it gets to the __init__.py file for dap, it skips it. This >causes the package to not work. It should also be installing a .pth file that makes it work, but if your final installation destination is not a 'site' directory (e.g. Python's site-packages directory), this will not work. > Anyone have an idea why it is doing this? Because system packaging tools like RPM et al do not like it when multiple packages install the same file. dap.plugins and dap.responses are "namespace packages", which means that other projects can install modules in them. Those modules mustn't install an __init__.py, as it would overwrite the one supplied by pydap. The setuptools solution to this problem is to never install an __init__.py at all for such packages, when used with a packaging tool (which is implied by the use of --root). Instead, setuptools generates a uniquely-named .pth file for each project, that sets up the namespace package at runtime. Look for a .pth file being placed in your /sw/src/fink.build/root-dap-py24-2.2.5.7-1/sw/lib/python2.4/site-packages/ directory. The problem is that if you then install the package somewhere other than /sw/lib/python2.4/site-packages, or use the -S option to Python at runtime, it may not work. >This install proceedure works well with other packages in mac osx/fink. I >also see the behavior without the --single-version-externally-managed flag. That's because --root implies --single-version-externally-managed. From pje at telecommunity.com Thu Jan 4 22:44:07 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 04 Jan 2007 16:44:07 -0500 Subject: [Distutils] [setuptools] egg_info and install_requires In-Reply-To: <23480D85-18E8-4456-87C4-0D7A12C7583B@mac.com> Message-ID: <5.1.1.6.0.20070104164318.04372fd8@sparrow.telecommunity.com> At 04:24 PM 1/2/2007 +0100, Ronald Oussoren wrote: >Hi, > >I have a setup.py that has some projects in the install_requires list >that setuptools cannot find yet (I haven't build the eggs yet) and >noticed something odd. When there is not yet an .egg-info directory >for my project everything is fine, I can run distutils commands >(except install of course). > >However, when the .egg-info directory is present all distutils >commands fail, including the egg_info command, because I'm refering >to a non-existing distribution. Is this supposed to happen? I'm not sure I understand the question. Could you provide more details, such as a link to the source, and what commands exactly are needed to produce the problem? Thanks. From jim at zope.com Thu Jan 4 23:10:55 2007 From: jim at zope.com (Jim Fulton) Date: Thu, 4 Jan 2007 17:10:55 -0500 Subject: [Distutils] tests_require and test_suite in egg-info In-Reply-To: <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> Message-ID: <87053F31-4170-4B0C-877E-F6B73257BC5D@zope.com> On Jan 4, 2007, at 4:37 PM, Phillip J. Eby wrote: > At 04:14 PM 1/4/2007 -0500, Jim Fulton wrote: > >> It appears that the values of the tests_require and test_suite >> keyword arguments isn't captured in egg-info. Am I missing >> something? Why would this information be omitted? > > Because the only existing tool that uses that information (the test > command) can't be run without the setup.py, which isn't installed > with the egg. uh huh. > In any case, many projects treat tests as development tools only, > rather than installing them with the project's main code. Many projects like to run tests even of packages they use, especially if there is a problem. >> I would like to >> use it when generating test runners in zc.buildout. > > Well, you could add an egg_info plugin writer to do it, or you > could add a custom setuptools command like "create_test_runners" > and run setup.py on it. There are probably also other ways I > haven't thought of. :) I can't do any of this if all I have is an egg. I want to be able to be able to run the tests that come with eggs. I suggest that it would be better for setuptools to record this information by default. Then again, I think that these are duplicative of other features. For example, tests_require could just be handled by an entry in extras_require for a "tests" extra and test suites could be handled as entry points. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jan 5 00:44:36 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 04 Jan 2007 18:44:36 -0500 Subject: [Distutils] tests_require and test_suite in egg-info In-Reply-To: <87053F31-4170-4B0C-877E-F6B73257BC5D@zope.com> References: <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070104183807.037dddb8@sparrow.telecommunity.com> At 05:10 PM 1/4/2007 -0500, Jim Fulton wrote: >I can't do any of this if all I have is an egg. I want to be able to >be able to run the >tests that come with eggs. Ah. >I suggest that it would be better for setuptools to record this >information by default. > >Then again, I think that these are duplicative of other features. >For example, >tests_require could just be handled by an entry in extras_require for >a "tests" >extra and test suites could be handled as entry points. The setuptools test command has a broader mission than just that, but having entry points for tests to be run by tools like zc.buildout is certainly a good idea; I suggest you create and publicize a de facto standard for them. The tricky part is that unless there is some way for setuptools to tell that: 1. the test suite is included, and 2. it can be adequately described as an entry point or collection thereof then I don't see a way to automatically map from existing test information to entry points. In particular, note that setuptools allows custom loaders to essentially usurp the entire standard testing process. Although testing is intended to be automatable, and have a common access point ("setup.py test"), the output of such test and the required environment (GUI, CLI, interactivity?) for them is not defined. So, trying to build a higher-level harness on the basis of what setuptools allows for is probably not a good idea here. However, the reverse is possible: if you define a standard for these things that the entry points must conform to, then I can make setuptools support *that* standard, and even recommend it in the documentation as the preferred way to specify a test target. Does that make sense? From kumar.mcmillan at gmail.com Fri Jan 5 05:22:31 2007 From: kumar.mcmillan at gmail.com (Kumar McMillan) Date: Thu, 4 Jan 2007 22:22:31 -0600 Subject: [Distutils] setuptools patch for egg_info.get_svn_revision; max() doesn't like an empty list Message-ID: hi folks. I looked high and low for an official spot to submit setuptools patches but just found this mailing list. Please accept my apologies in advance if this is inconvenient. I found that when running setup.py commands on an svn checkout while having just added a directory, but not committed it yet, I would get "ValueError: max() arg is an empty sequence" from around line 227 of egg_info.py. This seems easily fixable by checking the length of all_revs before handing it to max() and the patch attached solves the problem I was having. After applying the patch to the source, `python setup.py test` ran ok. thanks, Kumar -------------- next part -------------- A non-text attachment was scrubbed... Name: setuptools.r52354.get_svn_revision.patch Type: application/octet-stream Size: 935 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070104/6d86dd68/attachment.obj From pje at telecommunity.com Fri Jan 5 05:51:59 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 04 Jan 2007 23:51:59 -0500 Subject: [Distutils] setuptools patch for egg_info.get_svn_revision; max() doesn't like an empty list In-Reply-To: Message-ID: <5.1.1.6.0.20070104235128.027918d8@sparrow.telecommunity.com> At 10:22 PM 1/4/2007 -0600, Kumar McMillan wrote: >hi folks. >I looked high and low for an official spot to submit setuptools >patches but just found this mailing list. Please accept my apologies >in advance if this is inconvenient. Thanks; but a fix for this is already in the SVN trunk; you can get it with: easy_install setuptools==dev06 From ronaldoussoren at mac.com Fri Jan 5 08:06:17 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 5 Jan 2007 08:06:17 +0100 Subject: [Distutils] [setuptools] egg_info and install_requires In-Reply-To: <5.1.1.6.0.20070104164318.04372fd8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070104164318.04372fd8@sparrow.telecommunity.com> Message-ID: <30ABBD8E-1690-484C-BF26-D1CD351D3EA4@mac.com> On 4 Jan, 2007, at 22:44, Phillip J. Eby wrote: > At 04:24 PM 1/2/2007 +0100, Ronald Oussoren wrote: >> Hi, >> >> I have a setup.py that has some projects in the install_requires list >> that setuptools cannot find yet (I haven't build the eggs yet) and >> noticed something odd. When there is not yet an .egg-info directory >> for my project everything is fine, I can run distutils commands >> (except install of course). >> >> However, when the .egg-info directory is present all distutils >> commands fail, including the egg_info command, because I'm refering >> to a non-existing distribution. Is this supposed to happen? > > I'm not sure I understand the question. Could you provide more > details, such as a link to the source, and what commands exactly > are needed to produce the problem? Thanks. I'll see if I can whip up an example, a link to the source is not possible. What I thought to be a minimal example of the problem turns out to work correctly, I'll let you know when I find out what was wrong. A hopefull clearer explanation of the problem is that I have an empty package with a setup.py like this: setup( name="mypkg", version="1.0", setup_requires=[ 'docutils', ], install_requires=[ "i_dont_exist==1.0", "i_dont_exist_either==1.0", ], packages = [], ) The idea behind this is that the user will be able to use 'easy_install mypkg' and then will automaticly get the real software. The problem I ran into is that 'python setup.py bdist_egg' worked fine the first time, but the second time setuptools complained that it couldn't find a suitable distribution, e.g.: Couldn't find index page for 'i_dont_exist' (maybe misspelled?) No local packages or download links found for i_dont_exist Traceback (most recent call last): .... distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('i_dont_exist') As I mentioned before the setup.py int this message does NOT suffer from this problem. .... never mind, I found the actual problem, I had introduced a circular dependency through setup_requires. Sorry about bothering you, Ronald > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070105/cc648373/attachment-0001.bin From jim at zope.com Fri Jan 5 14:27:54 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 05 Jan 2007 08:27:54 -0500 Subject: [Distutils] tests_require and test_suite in egg-info In-Reply-To: <5.1.1.6.0.20070104183807.037dddb8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> <5.1.1.6.0.20070104163331.027f38b8@sparrow.telecommunity.com> <5.1.1.6.0.20070104183807.037dddb8@sparrow.telecommunity.com> Message-ID: <459E525A.30408@zope.com> Phillip J. Eby wrote: > At 05:10 PM 1/4/2007 -0500, Jim Fulton wrote: >> I can't do any of this if all I have is an egg. I want to be able to >> be able to run the >> tests that come with eggs. > > Ah. > > >> I suggest that it would be better for setuptools to record this >> information by default. >> >> Then again, I think that these are duplicative of other features. >> For example, >> tests_require could just be handled by an entry in extras_require for >> a "tests" >> extra and test suites could be handled as entry points. > > The setuptools test command has a broader mission than just that, Can you summarize the mission of the test command? AFAICT, it boils down to running tests. > but > having entry points for tests to be run by tools like zc.buildout is > certainly a good idea; Note that zc.buildout wouldn't run the test. A buildout recipe would create a test runner script that can run the tests. > I suggest you create and publicize a de facto > standard for them. OK. I'll propose something in a separate note. > The tricky part is that unless there is some way for setuptools to tell > that: > > 1. the test suite is included, and > 2. it can be adequately described as an entry point or collection thereof > > then I don't see a way to automatically map from existing test > information to entry points. > > In particular, note that setuptools allows custom loaders to essentially > usurp the entire standard testing process. Although testing is intended > to be automatable, and have a common access point ("setup.py test"), the > output of such test and the required environment (GUI, CLI, > interactivity?) for them is not defined. So, trying to build a > higher-level harness on the basis of what setuptools allows for is > probably not a good idea here. OK. I'm convinced. > However, the reverse is possible: if you define a standard for these > things that the entry points must conform to, then I can make setuptools > support *that* standard, and even recommend it in the documentation as > the preferred way to specify a test target. > > Does that make sense? Yup. Very much. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Fri Jan 5 14:46:44 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 05 Jan 2007 08:46:44 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs Message-ID: <459E56C4.5090303@zope.com> Here is a rough draft proposal for declaring tests in eggs: Introduction ============ Software packages should have automated tests. Consumers of packages will often want to run these tests. Tools should be able to do this automatically. This proposal seeks to provide a way for automated tools to discover tests in distributions, including eggs, so that tests can be run or so that test runners can be automatically created to run the tests. Proposal ======== This proposal aims to be extremely simple. It has 2 parts: 1. A 'test_suite' entry point is defined. An egg can provide zero or more test_suite entry points. These entry points will define callable objects that can be called without arguments and that return unittest test suites. 2. An optional 'tests' extra is defined. When creating test runners or dynamically loading distributions to load tests, any distributions listed in extra requires for the 'tests' extra shall be included in the working set for the test runner. Thoughts? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From schwehr at gmail.com Fri Jan 5 15:05:04 2007 From: schwehr at gmail.com (Kurt Schwehr) Date: Fri, 5 Jan 2007 09:05:04 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> References: <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> Message-ID: <37de72610701050605j674c07b0q168ff03be2554580@mail.gmail.com> Thanks for the reply, but I don't really understand your response. I don't know why multiple packages would install the same __init__.py file. I think that would be an error to do so on my part as a packager. Maybe I am missing the point. Here is some more info. Maybe this enough to show what I am doing wrong. This install does not work (e.g. "import dap" fails): %p/bin/python%type_raw[python] setup.py install --root=%d --single-version-externally-managed I do NOT get this file installed. /sw/lib/python2.5/site-packages/dap/__init__.py But this does work (as in the dap module works "import dap" and "import dap.client" but succeed and function correctly): %p/bin/python%type_raw[python] setup.py install --root=%d --single-version-externally-managed cp dap/__init__.py* %i/lib/python%type_raw[python]/site-packages/dap/ I then end up with /sw/lib/python2.5/site-packages/dap/__init__.py Here is what the pth looks like. Is there something wrong with it then? cat /sw/lib/python2.5/site-packages/dap-2.2.5.7-py2.5-nspkg.pth import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('dap', 'plugins')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('dap.plugins',new.module('dap.plugins')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('dap', 'responses')); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('dap.responses',new.module(' dap.responses')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) If I read that right, it is not letting python know about the dap module, so that I can't do something like "import dap.client". Does this mean that the setup.py is not configured correctly to include a "dap"? Thanks! -kurt On 1/4/07, Phillip J. Eby wrote: > > At 10:47 AM 1/3/2007 -0500, Kurt Schwehr wrote: > >Hi All, > > > >I am working on packaging pydap and am getting some strange behavior. > > > >Here is the install phase... > > > >/sw/bin/python2.4 setup.py install > >--root=/sw/src/fink.build/root-dap-py24-2.2.5.7-1 > >--single-version-externally-managed > > > > > >And when it gets to the __init__.py file for dap, it skips it. This > >causes the package to not work. > > It should also be installing a .pth file that makes it work, but if your > final installation destination is not a 'site' directory (e.g. Python's > site-packages directory), this will not work. > > > > Anyone have an idea why it is doing this? > > Because system packaging tools like RPM et al do not like it when multiple > packages install the same file. dap.plugins and dap.responses are > "namespace packages", which means that other projects can install modules > in them. Those modules mustn't install an __init__.py, as it would > overwrite the one supplied by pydap. The setuptools solution to this > problem is to never install an __init__.py at all for such packages, when > used with a packaging tool (which is implied by the use of --root). > > Instead, setuptools generates a uniquely-named .pth file for each project, > that sets up the namespace package at runtime. Look for a .pth file being > placed in your > > /sw/src/fink.build/root-dap-py24-2.2.5.7-1 > /sw/lib/python2.4/site-packages/ > > directory. The problem is that if you then install the package somewhere > other than /sw/lib/python2.4/site-packages, or use the -S option to Python > at runtime, it may not work. > > > >This install proceedure works well with other packages in mac > osx/fink. I > >also see the behavior without the --single-version-externally-managed > flag. > > That's because --root implies --single-version-externally-managed. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070105/140a6898/attachment.html From pje at telecommunity.com Fri Jan 5 17:43:46 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 11:43:46 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <37de72610701050605j674c07b0q168ff03be2554580@mail.gmail.co m> References: <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.com> At 09:05 AM 1/5/2007 -0500, Kurt Schwehr wrote: >Here is what the pth looks like. Is there something wrong with it then? > >cat /sw/lib/python2.5/site-packages/dap-2.2.5.7-py2.5-nspkg.pth >import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], >*('dap', 'plugins')); ie = os.path.exists(os.path.join (p,'__init__.py')); >m = not ie and >sys.modules.setdefault('dap.plugins',new.module('dap.plugins')); mp = (m >or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p) >import sys,new,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], >*('dap', 'responses')); ie = >os.path.exists(os.path.join(p,'__init__.py')); m = not ie and >sys.modules.setdefault(' dap.responses',new.module('dap.responses')); mp = >(m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and >mp.append(p) > >If I read that right, it is not letting python know about the dap module, >so that I can't do something like "import dap.client". Does this mean >that the setup.py is not configured correctly to include a "dap"? Yes, you're right, that is exactly the problem. There are two possible fixes: 1. add 'dap' to the namespace_packages setting in the setup.py 2. change setuptools to figure out that this is needed Since setuptools already knows that it shouldn't include the __init__ for dap, it can reasonably be considered a setuptools bug that it doesnt' generate the .pth correctly for that case, so I will fix that. In the meantime, as a workaround, adding 'dap' to the namespace_packages setting will allow you to proceed. Thanks for your patience and help in identifying the actual bug. From davidf at sjsoft.com Fri Jan 5 16:36:08 2007 From: davidf at sjsoft.com (David Fraser) Date: Fri, 05 Jan 2007 17:36:08 +0200 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E56C4.5090303@zope.com> References: <459E56C4.5090303@zope.com> Message-ID: <459E7068.4060402@sjsoft.com> Jim Fulton wrote: > Here is a rough draft proposal for declaring tests in eggs: > > Introduction > ============ > > Software packages should have automated tests. Consumers of > packages will often want to run these tests. Tools should be able to > do this automatically. This proposal seeks to provide a way for > automated tools to discover tests in distributions, including eggs, so > that tests can be run or so that test runners can be automatically > created to run the tests. > > Proposal > ======== > > This proposal aims to be extremely simple. It has 2 parts: > > 1. A 'test_suite' entry point is defined. An egg can provide zero or > more test_suite entry points. These entry points will define > callable objects that can be called without arguments and that > return unittest test suites. > How would this work if for example, you're using an alternative testing framework (like py.test) for your test? It would be nice to be able to bootstrap it :-) > 2. An optional 'tests' extra is defined. When creating test runners > or dynamically loading distributions to load tests, any > distributions listed in extra requires for the 'tests' extra shall > be included in the working set for the test runner. > Great, so at least the testing framework could be declared as a dependency David From jim at zope.com Fri Jan 5 18:04:46 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 05 Jan 2007 12:04:46 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E7068.4060402@sjsoft.com> References: <459E56C4.5090303@zope.com> <459E7068.4060402@sjsoft.com> Message-ID: <459E852E.7020204@zope.com> David Fraser wrote: > Jim Fulton wrote: >> Here is a rough draft proposal for declaring tests in eggs: >> >> Introduction >> ============ >> >> Software packages should have automated tests. Consumers of >> packages will often want to run these tests. Tools should be able to >> do this automatically. This proposal seeks to provide a way for >> automated tools to discover tests in distributions, including eggs, so >> that tests can be run or so that test runners can be automatically >> created to run the tests. >> >> Proposal >> ======== >> >> This proposal aims to be extremely simple. It has 2 parts: >> >> 1. A 'test_suite' entry point is defined. An egg can provide zero or >> more test_suite entry points. These entry points will define >> callable objects that can be called without arguments and that >> return unittest test suites. >> > How would this work if for example, you're using an alternative testing > framework (like py.test) for your test? I'm not familiar with py.test. I guess it's not based on unittest. Why? Couldn't it at least have a unittest wrapper, like the one I wrote for doctest? Even though I use doctest almost exclusively, I view unittest as a common API that various testing frameworks can and should play with. I certainly think there should be some common API like that and see unittest as the incumbant. > It would be nice to be able to bootstrap it :-) What do you mean by that? >> 2. An optional 'tests' extra is defined. When creating test runners >> or dynamically loading distributions to load tests, any >> distributions listed in extra requires for the 'tests' extra shall >> be included in the working set for the test runner. >> > Great, so at least the testing framework could be declared as a dependency My assumption (and my use case) is that you'd run whatever test runner you want and tell it to run tests from some given eggs. So the test runner would already be loaded. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jan 5 18:06:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 12:06:12 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E56C4.5090303@zope.com> Message-ID: <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote: >Here is a rough draft proposal for declaring tests in eggs: > >Introduction >============ > >Software packages should have automated tests. Consumers of >packages will often want to run these tests. Tools should be able to >do this automatically. This proposal seeks to provide a way for >automated tools to discover tests in distributions, including eggs, so >that tests can be run or so that test runners can be automatically >created to run the tests. > >Proposal >======== > >This proposal aims to be extremely simple. It has 2 parts: > >1. A 'test_suite' entry point is defined. An egg can provide zero or > more test_suite entry points. These entry points will define > callable objects that can be called without arguments and that > return unittest test suites. > >2. An optional 'tests' extra is defined. When creating test runners > or dynamically loading distributions to load tests, any > distributions listed in extra requires for the 'tests' extra shall > be included in the working set for the test runner. > >Thoughts? Point #2 is unnecessary, since individual entry points can list extras in square brackets following the module/attribute information. When loading an entry point, these extras automatically get require()'d. Conversely, if the test runner wants to manage the loading process, it can simply inspect the entry point object to determine the names of the extras. Regarding point #1, I'm not sure this is enought to define what's necessary. For example, it should perhaps be stated that the entry point must be in code that will be installed by either the distribution itself, or that is included in the code provided by the entry point's extras. Also, an egg can't provide more than one entry point with the same name, so rather than a 'test_suite' entry point, we probably want an entry point *group*, perhaps something like 'installed.test_suites'. Next, there needs to be reasonable support for dynamic test discovery, such as setuptools' own ScanningLoader. I would suggest that there be an optional entry point for the test loader class to be used to process the other entry points' target objects. The standard unittest protocol for loadTestsFromName() takes two arguments: a name and an object. Passing an empty string for the name, and the object loaded by the test suite entry point, suffices to enable normal behavior for loaders such as ScanningLoader, and I believe 'nose' as well as any other well-behaved unittest extensions. However, I think the spec should try to define what "well-behaved" means in terms of I/O, result reporting, etc. (Obviously, one requirement is that the test loader must be able to take an empty name string and an object in its loadTestsFromName() method, and return a test suite.) From pje at telecommunity.com Fri Jan 5 18:13:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 12:13:27 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E7068.4060402@sjsoft.com> References: <459E56C4.5090303@zope.com> <459E56C4.5090303@zope.com> Message-ID: <5.1.1.6.0.20070105120715.04cf7bc8@sparrow.telecommunity.com> At 05:36 PM 1/5/2007 +0200, David Fraser wrote: >How would this work if for example, you're using an alternative testing >framework (like py.test) for your test? If I understand correctly, py.test simply won't be usable for this scenario, because it assumes tests are files first and modules second, if at all. It's not equipped to discover tests that are inside eggs, which is the whole point of this proposal. Integrating py.test with the "setup.py test" command, on the other hand is possible and relatively simple; just create a unittest-compatible test suite object that wraps py.test, and a test loader class that grabs the necessary arguments from the test_suite argument string. No modifications to setuptools are required, you just provide appropriate values for the test_suite and test_loader arguments to setup(). But the proposal being discussed is for tests that are actually *installed* somewhere, and thus might be in a zipfile. >Great, so at least the testing framework could be declared as a dependency For "setup.py test", this is already possible via the 'tests_require' argument to setup(). What's being discussed here is a standard for *installed* tests -- that is, tests that a developer has chosen to make part of their projects's installed code. The existing "setup.py test" subsystem requires a *source* distribution, not installed code. From jim at zope.com Fri Jan 5 18:24:10 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 05 Jan 2007 12:24:10 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> References: <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> Message-ID: <459E89BA.5010608@zope.com> Phillip J. Eby wrote: > At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote: > >> Here is a rough draft proposal for declaring tests in eggs: >> >> Introduction >> ============ >> >> Software packages should have automated tests. Consumers of >> packages will often want to run these tests. Tools should be able to >> do this automatically. This proposal seeks to provide a way for >> automated tools to discover tests in distributions, including eggs, so >> that tests can be run or so that test runners can be automatically >> created to run the tests. >> >> Proposal >> ======== >> >> This proposal aims to be extremely simple. It has 2 parts: >> >> 1. A 'test_suite' entry point is defined. An egg can provide zero or >> more test_suite entry points. These entry points will define >> callable objects that can be called without arguments and that >> return unittest test suites. >> >> 2. An optional 'tests' extra is defined. When creating test runners >> or dynamically loading distributions to load tests, any >> distributions listed in extra requires for the 'tests' extra shall >> be included in the working set for the test runner. >> >> Thoughts? > > Point #2 is unnecessary, since individual entry points can list extras > in square brackets following the module/attribute information. When > loading an entry point, these extras automatically get require()'d. > Conversely, if the test runner wants to manage the loading process, it > can simply inspect the entry point object to determine the names of the > extras. Ah, good point. Man, setuptools is soooooo rich. > Regarding point #1, I'm not sure this is enought to define what's > necessary. For example, it should perhaps be stated that the entry > point must be in code that will be installed by either the distribution > itself, or that is included in the code provided by the entry point's > extras. OK > Also, an egg can't provide more than one entry point with the same name, > so rather than a 'test_suite' entry point, we probably want an entry > point *group*, perhaps something like 'installed.test_suites'. That's what I meant to say. I'll clarify. > Next, there needs to be reasonable support for dynamic test discovery, > such as setuptools' own ScanningLoader. I would suggest that there be > an optional entry point for the test loader class to be used to process > the other entry points' target objects. The standard unittest protocol > for loadTestsFromName() takes two arguments: a name and an object. > Passing an empty string for the name, and the object loaded by the test > suite entry point, suffices to enable normal behavior for loaders such > as ScanningLoader, and I believe 'nose' as well as any other > well-behaved unittest extensions. > > However, I think the spec should try to define what "well-behaved" means > in terms of I/O, result reporting, etc. (Obviously, one requirement is > that the test loader must be able to take an empty name string and an > object in its loadTestsFromName() method, and return a test suite.) Why can't an entry point invoke a test loader itself? This seems much simpler and more straightforward to me. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jan 5 18:35:22 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 12:35:22 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E89BA.5010608@zope.com> References: <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> At 12:24 PM 1/5/2007 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >>At 08:46 AM 1/5/2007 -0500, Jim Fulton wrote: >>>2. An optional 'tests' extra is defined. When creating test runners >>> or dynamically loading distributions to load tests, any >>> distributions listed in extra requires for the 'tests' extra shall >>> be included in the working set for the test runner. >>> >>>Thoughts? >>Point #2 is unnecessary, since individual entry points can list extras in >>square brackets following the module/attribute information. When loading >>an entry point, these extras automatically get require()'d. >>Conversely, if the test runner wants to manage the loading process, it >>can simply inspect the entry point object to determine the names of the extras. > >Ah, good point. Man, setuptools is soooooo rich. Rich requirements make rich design. See my recent "Where Zope Leads, Python Follows" blog post. :) >Why can't an entry point invoke a test loader itself? >This seems much simpler and more straightforward to me. Because that requires you to write code for something that can adequately be expressed through an existing configuration mechanism. *And* you have to write that code in every project. If you're using setuptools' ScanningLoader (which is the default for the "test" command, and which I believe should be the default for this proposal as well), you automatically get all TestCase subclasses in all submodules of the package you target, and if any module has an 'additional_tests()' function, it's called to get a suite that's added in. This lets you just make your entry point a "whatever.tests" package, without writing any extra code (unless you want to throw in some doctests via a few 'additional_tests()' functions). See the setuptools documentation for the 'test_suite' and 'test_loader' arguments to setup() for the full details. Oh, and the 'nose' package is a unittest-compatible extended testing framework that is designed to also work as a 'test_loader' for setuptools. So, between ScanningLoader and 'nose', that would be two already-existing ways of specifying tests that would work with the new proposal, if you could specify a loader entry point. And it would be good if the default loader was compatible with ScanningLoader's test discovery features. (I'm not sure if nose really works with eggs, though; ScanningLoader will discover tests in eggs as long as the source is included, but nose may be strictly file-based for all I know. Changing it probably wouldn't be too difficult, however.) From jim at zope.com Fri Jan 5 19:11:19 2007 From: jim at zope.com (Jim Fulton) Date: Fri, 05 Jan 2007 13:11:19 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> Message-ID: <459E94C7.9060101@zope.com> Phillip J. Eby wrote: ... >> Why can't an entry point invoke a test loader itself? >> This seems much simpler and more straightforward to me. > > Because that requires you to write code for something that can > adequately be expressed through an existing configuration mechanism. > *And* you have to write that code in every project. It's probably a couple of lines. Import a loader and call it. That doesn't seem like a big deal to me. Nice and explicit too. Besides, I expect most projects will have a single test suite (function) that could just be named directly. > If you're using setuptools' ScanningLoader (which is the default for the > "test" command, and which I believe should be the default for this > proposal as well), you automatically get all TestCase subclasses in all > submodules of the package you target, and if any module has an > 'additional_tests()' function, it's called to get a suite that's added > in. This lets you just make your entry point a "whatever.tests" > package, without writing any extra code (unless you want to throw in > some doctests via a few 'additional_tests()' functions). > > See the setuptools documentation for the 'test_suite' and 'test_loader' > arguments to setup() for the full details. I've looked at this and it is a bit too magic for my taste. It's also a heck of a lot to get one's head around. (This is true of setuptools in general.) Of course, it doesn't help much with our doctests and for our older unittest tests, we have lots of TestCase subclasses that aren't meant to be run directly, but are base classes for other TestCase classes. It appears that your loader would pick these up incorrectly. Of course, we have our own loader (of sorts) and it has it's own magic conventions. We would likely use that for our larger projects (e.g. ZODB) that have more than a few test suites. Importing some function and invoking it explicitly will not be the least bit onerous for those projects. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Fri Jan 5 19:28:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 13:28:45 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <459E94C7.9060101@zope.com> References: <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote: >Phillip J. Eby wrote: >... >>>Why can't an entry point invoke a test loader itself? >>>This seems much simpler and more straightforward to me. >>Because that requires you to write code for something that can adequately >>be expressed through an existing configuration mechanism. >>*And* you have to write that code in every project. > >It's probably a couple of lines. Import a loader and call it. >That doesn't seem like a big deal to me. Nice and explicit too. And it's enormously repetitive if you have a lot of projects. >Besides, I expect most projects will have a single test suite >(function) that could just be named directly. You expect wrongly. :) 'nose' and 'py.test' are popular *precisely because* they do NOT require this. Setuptools changed to emulate them by adding ScanningLoader, even though it originally did the "test suite function" thing you're proposing. See, I originally wrote all my test suites the way you're proposing, but I changed because this is one of those "convention beats configuration" situations. It's easier to just conform to a policy and have a tool that applies the policy. Different projects may have different policies, so providing an option for the loader, allows them to follow it. (And it made my life easier, too, as I began doing more small projects instead of a few large ones.) And, if you want your approach to be widely adopted, it would be best to allow projects to follow their own policies without requiring them to add any code, even if it's a one-liner. So, *technically*, there may be no reason to do this, but from a usability, friendliness, compatiblity, marketing, adoption, etc. format I don't think the standard will be successful without allowing this out. People often have heavy psychological investment in existing tools, and it is hundreds of times easier to rationalize adding a line of configuration to setup() than it is to rationalize adding two or three lines of code. One is merely a packaging change, the other is a programming change. From pje at telecommunity.com Fri Jan 5 20:41:46 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 14:41:46 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.co m> References: <37de72610701050605j674c07b0q168ff03be2554580@mail.gmail.co m> <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070105144042.0284ad80@sparrow.telecommunity.com> I've now fixed the bug below in both the 0.6 branch and the development trunk. You can get them by easy_installing either 'setuptools==dev06' or 'setuptools==dev', respectively. At 11:43 AM 1/5/2007 -0500, Phillip J. Eby wrote: >At 09:05 AM 1/5/2007 -0500, Kurt Schwehr wrote: > >If I read that right, it is not letting python know about the dap module, > >so that I can't do something like "import dap.client". Does this mean > >that the setup.py is not configured correctly to include a "dap"? > >Yes, you're right, that is exactly the problem. There are two possible fixes: > >1. add 'dap' to the namespace_packages setting in the setup.py > >2. change setuptools to figure out that this is needed > >Since setuptools already knows that it shouldn't include the __init__ for >dap, it can reasonably be considered a setuptools bug that it doesnt' >generate the .pth correctly for that case, so I will fix that. In the >meantime, as a workaround, adding 'dap' to the namespace_packages setting >will allow you to proceed. > >Thanks for your patience and help in identifying the actual bug. From srobertson at codeit.com Sat Jan 6 01:35:22 2007 From: srobertson at codeit.com (Scott Robertson) Date: Fri, 5 Jan 2007 16:35:22 -0800 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked Message-ID: <905d10ce0701051635h630e16c9udecbfb51059d6e97@mail.gmail.com> I'm trying to create a package that provides a setuptool command that will compile idl files when you run python setup.py build or python setup.pyinstall. I've figured out how to add an additional command (build_omniidl) which I can run with: python setup.py build_omniidl But I'm not sure how to wire it up so that python setup.py build automatically invokes build_omniidl for me. Basically I want build_omniidl to act just like build_python. Is this even possible with setuptools? If so anyone know where I can see an example? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070105/12ab7e30/attachment.html From pje at telecommunity.com Sat Jan 6 02:07:04 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 05 Jan 2007 20:07:04 -0500 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked In-Reply-To: <905d10ce0701051635h630e16c9udecbfb51059d6e97@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070105200402.042ac818@sparrow.telecommunity.com> At 04:35 PM 1/5/2007 -0800, Scott Robertson wrote: >I'm trying to create a package that provides a setuptool command that will >compile idl files when you run python setup.py build or python setup.py >install. > >I've figured out how to add an additional command (build_omniidl) which I >can run with: > python setup.py build_omniidl > >But I'm not sure how to wire it up so that > python setup.py build > >automatically invokes build_omniidl for me. Basically I want build_omniidl >to act just like build_python. > > >Is this even possible with setuptools? It's possible even with the distutils. What you need is to subclass the distutils 'build' command class, and then pass your subclass in via the 'cmdclass' dictionary argument to setup(). In setuptools 0.7, I plan to allow adding support for these kinds of extensions using entry points, so that subclassing isn't needed. >If so anyone know where I can see an example? Not right off, no. Check the distutils documentation; I seem to recall it at least has some information about subclassing and using cmdclass. From schwehr at gmail.com Sat Jan 6 21:46:32 2007 From: schwehr at gmail.com (Kurt Schwehr) Date: Sat, 6 Jan 2007 15:46:32 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.com> Message-ID: <37de72610701061246v1a5b7accudb47a0a34eab3f5a@mail.gmail.com> Sounds good, but I still am confused as to why there should not be a site-packages/dap/__init__.py in the installed lib area. For this package there isn't too much in there, but some packages include things like version numbers and defaults in their __init__.py. For dap, we would be leaving out the overview documentation for import dap;help(dap). If the case really is that there should not be a dap/__init__.py, where do I tell the author of the package to move that documentation such that help(dap) will still provide as much info? Thanks! -kurt > Yes, you're right, that is exactly the problem. There are two possible > fixes: > > 1. add 'dap' to the namespace_packages setting in the setup.py > > 2. change setuptools to figure out that this is needed > > Since setuptools already knows that it shouldn't include the __init__ for > dap, it can reasonably be considered a setuptools bug that it doesnt' > generate the .pth correctly for that case, so I will fix that. In the > meantime, as a workaround, adding 'dap' to the namespace_packages setting > will allow you to proceed. > > Thanks for your patience and help in identifying the actual bug. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070106/ae76dd52/attachment.htm From pje at telecommunity.com Sat Jan 6 22:05:47 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 06 Jan 2007 16:05:47 -0500 Subject: [Distutils] Skipping namespace file? In-Reply-To: <37de72610701061246v1a5b7accudb47a0a34eab3f5a@mail.gmail.co m> References: <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.com> <5.1.1.6.0.20070104163812.04a19690@sparrow.telecommunity.com> <5.1.1.6.0.20070105114135.02699cc0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070106160316.028dd688@sparrow.telecommunity.com> At 03:46 PM 1/6/2007 -0500, Kurt Schwehr wrote: >Sounds good, but I still am confused as to why there should not be a >site-packages/dap/__init__.py in the installed lib area. For this package >there isn't too much in there, but some packages include things like >version numbers and defaults in their __init__.py. Yes, but then they shouldn't declare them (implicitly or explicitly) as namespace packages. A namespace package, by definition, is one whose sole purpose is to provide a namespace for modules and subpackages. It cannot provide any contents of its own. > For dap, we would be leaving out the overview documentation for import > dap;help(dap). If the case really is that there should not be a > dap/__init__.py, where do I tell the author of the package to move that > documentation such that help(dap) will still provide as much info? There isn't any such place, I'm afraid. The alternative would be to use some other package as a base for plugins and responses, e.g. dap_ext.plugins and dap_ext.responses, so that 'dap' would not be a namespace package itself. From lutz_p at gmx.net Mon Jan 8 15:11:53 2007 From: lutz_p at gmx.net (=?iso-8859-1?Q?=22Lutz_P=E4like=22?=) Date: Mon, 08 Jan 2007 15:11:53 +0100 Subject: [Distutils] custom egg metadata Message-ID: <20070108141153.97340@gmx.net> I have some problems customizing the egg building process. I want to add a custom file to the egg metadata and the documentation says a egg_info.writers entry should work, but it looks like my definition is not evaluated. I tried to trace this down and found some more information. This is how my args to setup look like: {'entry_points': { 'distutils.setup_keywords': ['xyz_settings=setuptools.dist:assert_string_list'], 'egg_info.writers': ['xyz_settings.txt = setuptools.command.egg_info:write_arg'] }, 'name': u'XYZ_Project', 'description': 'My XYZ Project', 'package_dir': {'': 'data/project'}, 'packages': ['XYZ_Project'], 'version': '0.1' 'xyz_settings' : [ "Line 1", "Line 2" ] } Now if i step through the building process in the debugger i get the following: - distutils/dist.py:236: UserWarning: Unknown distribution option: 'xyz_settings' - if i step until 'setuptools/command/egg_info.py(169)run()' i see the following line: for ep in iter_entry_points('egg_info.writers') so this looks like the place where my 'egg_info.writers' definition is evaluated, but if i inspect iter_entry_points('egg_info.writers') i get the following: [ EntryPoint.parse('requires.txt = setuptools.command.egg_info:write_requirements'), EntryPoint.parse('PKG-INFO = setuptools.command.egg_info:write_pkg_info'), EntryPoint.parse('eager_resources.txt = setuptools.command.egg_info:overwrite_arg'), EntryPoint.parse('namespace_packages.txt = setuptools.command.egg_info:overwrite_arg'), EntryPoint.parse('top_level.txt = setuptools.command.egg_info:write_toplevel_names'), EntryPoint.parse('dependency_links.txt = setuptools.command.egg_info:overwrite_arg'), EntryPoint.parse('entry_points.txt = setuptools.command.egg_info:write_entries'), EntryPoint.parse('depends.txt = setuptools.command.egg_info:warn_depends_obsolete') ] There appears no definition of xyz_settings.txt ? Am i missing something here ? If i look closer inside iter_entry_points() i see the following code: for dist in self: entries = dist.get_entry_map(group) if i inspect the list of Distributions in self i get all the Distributions i installed via easy_install to my system but not the package i am currently building. So how is this supposed to work ? Should'nt the Distribution i am building be in this list? How are my entry points found then if not? Sorry if this is obvious, but i would be glad if somebody could give some more information here. Thanks, Lutz Paelike -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal f?r Modem und ISDN: http://www.gmx.net/de/go/smartsurfer From davidf at sjsoft.com Mon Jan 8 15:38:00 2007 From: davidf at sjsoft.com (David Fraser) Date: Mon, 08 Jan 2007 16:38:00 +0200 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <5.1.1.6.0.20070105120715.04cf7bc8@sparrow.telecommunity.com> References: <459E56C4.5090303@zope.com> <459E56C4.5090303@zope.com> <5.1.1.6.0.20070105120715.04cf7bc8@sparrow.telecommunity.com> Message-ID: <45A25748.9090903@sjsoft.com> Phillip J. Eby wrote: > At 05:36 PM 1/5/2007 +0200, David Fraser wrote: >> How would this work if for example, you're using an alternative testing >> framework (like py.test) for your test? > If I understand correctly, py.test simply won't be usable for this > scenario, because it assumes tests are files first and modules second, > if at all. It's not equipped to discover tests that are inside eggs, > which is the whole point of this proposal. > Integrating py.test with the "setup.py test" command, on the other > hand is possible and relatively simple; just create a > unittest-compatible test suite object that wraps py.test, and a test > loader class that grabs the necessary arguments from the test_suite > argument string. No modifications to setuptools are required, you > just provide appropriate values for the test_suite and test_loader > arguments to setup(). > > But the proposal being discussed is for tests that are actually > *installed* somewhere, and thus might be in a zipfile. So I need to go and code a test enumerator for py.test to discover tests within eggs (which is possible) >> Great, so at least the testing framework could be declared as a >> dependency > For "setup.py test", this is already possible via the 'tests_require' > argument to setup(). What's being discussed here is a standard for > *installed* tests -- that is, tests that a developer has chosen to > make part of their projects's installed code. The existing "setup.py > test" subsystem requires a *source* distribution, not installed code. OK great, thanks for clarifying... David From pje at telecommunity.com Mon Jan 8 16:54:34 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 08 Jan 2007 10:54:34 -0500 Subject: [Distutils] custom egg metadata In-Reply-To: <20070108141153.97340@gmx.net> Message-ID: <5.1.1.6.0.20070108104950.028a86a8@sparrow.telecommunity.com> At 03:11 PM 1/8/2007 +0100, Lutz P?like wrote: > There appears no definition of xyz_settings.txt ? > Am i missing something here ? Your custom writer has to be on sys.path in order to work, but your package_dir isn't on sys.path. You'll have to either install the writer as a separate egg and use setup_requires to ensure it gets on the path, or else move your package_dir to the setup.py directory. The general intention of the design, however, was for egg_info writers to live in a different egg than the one being built, and to be activated using setup_requires. Still, you *can* make it work, it's just that you'll have to run the egg_info command (or a command that runs egg_info) more than once in order to get your custom metadata written, because the first time the egg_info is written, your entry points won't be on disk yet. The second time, it will. From jjl at pobox.com Mon Jan 8 19:05:41 2007 From: jjl at pobox.com (John J Lee) Date: Mon, 8 Jan 2007 18:05:41 +0000 (GMT Standard Time) Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <45A25748.9090903@sjsoft.com> References: <459E56C4.5090303@zope.com> <459E56C4.5090303@zope.com> <5.1.1.6.0.20070105120715.04cf7bc8@sparrow.telecommunity.com> <45A25748.9090903@sjsoft.com> Message-ID: On Mon, 8 Jan 2007, David Fraser wrote: [...] > So I need to go and code a test enumerator for py.test to discover tests > within eggs (which is possible) [...] Or (for new code) use nose, which AIUI is intended to be pretty much py.test implemented within the unittest framework (plus whatever bits and pieces the nose authors wanted to change, it's true). John From pje at telecommunity.com Mon Jan 8 20:02:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 08 Jan 2007 14:02:27 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: References: <45A25748.9090903@sjsoft.com> <459E56C4.5090303@zope.com> <459E56C4.5090303@zope.com> <5.1.1.6.0.20070105120715.04cf7bc8@sparrow.telecommunity.com> <45A25748.9090903@sjsoft.com> Message-ID: <5.1.1.6.0.20070108135651.028ad330@sparrow.telecommunity.com> At 06:05 PM 1/8/2007 +0000, John J Lee wrote: >On Mon, 8 Jan 2007, David Fraser wrote: >[...] > > So I need to go and code a test enumerator for py.test to discover tests > > within eggs (which is possible) >[...] > >Or (for new code) use nose, which AIUI is intended to be pretty much >py.test implemented within the unittest framework (plus whatever bits and >pieces the nose authors wanted to change, it's true). I'm not sure whether nose handles test discovery in eggs, but I believe that, like py.test, it has pluggable enumeration capabilities. What would be nice is if we could get some folks working on py.test and nose to chime in on the proposal, to make sure that it's sound. At the moment, the proposal (or at least my suggested version of it) allows for: * specification of one "loader" for all tests in an egg * specification of zero or more entry points for locating tests The only way to provide any configuration to the loader is via attributes of the entry points. For example, if the entry point is a module, you could set your options location via attributes of the module. The loader must support a .loadTestFromName(name, obj) call, where name will be an empty string, and obj will be the object specified by a test entry point. The return value must be a unittest.TestSuite. And it must be possible to call the loader multiple times with different 'obj' values. From ronaldoussoren at mac.com Tue Jan 9 09:51:30 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 9 Jan 2007 09:51:30 +0100 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked In-Reply-To: <5.1.1.6.0.20070105200402.042ac818@sparrow.telecommunity.com> References: <5.1.1.6.0.20070105200402.042ac818@sparrow.telecommunity.com> Message-ID: <41D758FD-B18D-4EB1-8817-F66126130E68@mac.com> On 6 Jan, 2007, at 2:07, Phillip J. Eby wrote: > At 04:35 PM 1/5/2007 -0800, Scott Robertson wrote: >> I'm trying to create a package that provides a setuptool command >> that will >> compile idl files when you run python setup.py build or python >> setup.py >> install. >> >> I've figured out how to add an additional command (build_omniidl) >> which I >> can run with: >> python setup.py build_omniidl >> >> But I'm not sure how to wire it up so that >> python setup.py build >> >> automatically invokes build_omniidl for me. Basically I want >> build_omniidl >> to act just like build_python. >> >> >> Is this even possible with setuptools? > > It's possible even with the distutils. What you need is to > subclass the > distutils 'build' command class, and then pass your subclass in via > the > 'cmdclass' dictionary argument to setup(). > > In setuptools 0.7, I plan to allow adding support for these kinds of > extensions using entry points, so that subclassing isn't needed. Another alternative is to tweak distutils.commands.build.build.sub_commands, which is more convenient than subclassing. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 3562 bytes Desc: not available Url : http://mail.python.org/pipermail/distutils-sig/attachments/20070109/124075ea/attachment.bin From lutz_p at gmx.net Tue Jan 9 12:34:34 2007 From: lutz_p at gmx.net (=?iso-8859-1?Q?=22Lutz_P=E4like=22?=) Date: Tue, 09 Jan 2007 12:34:34 +0100 Subject: [Distutils] custom egg metadata Message-ID: <20070109113434.57780@gmx.net> > Your custom writer has to be on sys.path in order to work, but your > package_dir isn't on sys.path. Well this is the case, since i want to use a general setuptools writer and need no fancy custom writer (yet). > You'll have to either install the writer as a separate egg and use > setup_requires to ensure it gets on the path, or else move your > package_dir to the setup.py directory. Ok. But the writer just needs to be importable right? there is no real need to have the package_dir directly alongside the setup.py ? I have only a single setup.py, but lots of subprojects that i want to keep in subfolders. And there are no setup.py in the project subfolders but everything is controlled through the single setup.py and a config file within each project. > The general intention of the design, however, was for egg_info writers to > live in a different egg than the one being built, and to be activated > using setup_requires. Ok that's fine. > Still, you *can* make it work, it's just that you'll have to run the > egg_info command (or a command that runs egg_info) more than once in > order to get your custom metadata written, because the first time the > egg_info is written, your entry points won't be on disk yet. The second > time, it will. Ok i tried now what you suggested and it worked as you described if i split the build process in 2 steps. However it does not if i want to put the two steps in a single script, which indicates that some data is persistent and the freshly generated meta information is not yet respected. I tried to set pkg_resources.working_set , but it does not seem to have the necessary impact for the bdist_egg command. I guess the working_set instance is imported at an earlier stage into setuptools and thus does'nt get updated. I think the main problems i got is because i want to use setuptools from an api perspective and not from a tool perspective. Finally i want to integrate the setup script into a gui and make the building accessible to non-developers (this is a necessity). And because i want to catch exceptions and all that i don't like to use something like os.system() to launch different steps of the build. Maybe you can give me some more hints why this fails. At present i invoke the egg_info and the bdist_egg build step by setting the 'script_args' parameter for each step and call setup(). As already mentioned this fails. Regards, Lutz Paelike -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal f?r Modem und ISDN: http://www.gmx.net/de/go/smartsurfer From lutz_p at gmx.net Tue Jan 9 15:51:41 2007 From: lutz_p at gmx.net (=?iso-8859-1?Q?=22Lutz_P=E4like=22?=) Date: Tue, 09 Jan 2007 15:51:41 +0100 Subject: [Distutils] custom egg metadata Message-ID: <20070109145141.178990@gmx.net> Ok i solved the problem now. I expanded sys.path to reflect my environment but the changes did not have any effect on setuptools. Now i noticed that i have to use pkg_resources.working_set.add_entry() to let setuptools take notice of the change. That did the trick. Thanks for your time. Cheers, Lutz -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal f?r Modem und ISDN: http://www.gmx.net/de/go/smartsurfer From lutz_p at gmx.net Tue Jan 9 15:42:58 2007 From: lutz_p at gmx.net (Lutz Paelike) Date: Tue, 09 Jan 2007 15:42:58 +0100 Subject: [Distutils] custom egg metadata Message-ID: <45A3A9F2.307@gmx.net> Ok i solved the problem now. I expanded sys.path to reflect my environment but the changes did not have any effect on setuptools. Now i noticed that i have to use pkg_resources.working_set.add_entry() to let setuptools take notice of the change. That did the trick. Thanks for your time. Cheers, Lutz From pje at telecommunity.com Tue Jan 9 16:57:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 10:57:27 -0500 Subject: [Distutils] custom egg metadata In-Reply-To: <20070109113434.57780@gmx.net> Message-ID: <5.1.1.6.0.20070109104744.0289ec30@sparrow.telecommunity.com> At 12:34 PM 1/9/2007 +0100, Lutz P?like wrote: > > Your custom writer has to be on sys.path in order to work, but your > > package_dir isn't on sys.path. > >Well this is the case, since i want to use a general setuptools writer and >need no fancy custom writer (yet). > > > You'll have to either install the writer as a separate egg and use > > setup_requires to ensure it gets on the path, or else move your > > package_dir to the setup.py directory. > >Ok. But the writer just needs to be importable right? It needs to be an *egg* on sys.path, meaning that both the source and the .egg-info are there. > there is no real need to have the package_dir directly alongside the > setup.py ? >I have only a single setup.py, but lots of subprojects that i want to keep >in subfolders. And there are no setup.py in the project subfolders >but everything is controlled through the single setup.py and a config file >within each project. That's not really a supported arrangement; note that you won't be able to make proper source distributions, for example. (Source distributions must contain a setup.py to be installable.) >I tried to set pkg_resources.working_set , but it does not seem to have >the necessary impact for the bdist_egg command. >I guess the working_set instance is imported at an earlier stage into >setuptools and thus does'nt get updated. Correct. Also, your package_dir would have to be on sys.path. >I think the main problems i got is because i want to use setuptools from >an api perspective and not from a tool perspective. >Finally i want to integrate the setup script into a gui and make the >building accessible to non-developers (this is a necessity). >And because i want to catch exceptions and all that i don't like to use >something like os.system() to launch different steps of >the build. > >Maybe you can give me some more hints why this fails. To be blunt, the distutils doesn't really support what you're trying to do. And setuptools inherits certain of its limitations, like the fact that it is built to assume that there is a setup.py in the current directory and the current directory is the root of a project. >At present i invoke the egg_info and the bdist_egg build step by setting >the 'script_args' parameter for each step and call >setup(). As already mentioned this fails. You would need to run egg_info, put the package_dir on sys.path, require() the new distribution, then run egg_info a second time (or bdist_egg, which calls egg_info). From pje at telecommunity.com Tue Jan 9 16:59:24 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 10:59:24 -0500 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked In-Reply-To: <41D758FD-B18D-4EB1-8817-F66126130E68@mac.com> References: <5.1.1.6.0.20070105200402.042ac818@sparrow.telecommunity.com> <5.1.1.6.0.20070105200402.042ac818@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070109105751.02895dd0@sparrow.telecommunity.com> At 09:51 AM 1/9/2007 +0100, Ronald Oussoren wrote: >Another alternative is to tweak >distutils.commands.build.build.sub_commands, which is more convenient >than subclassing. But will crash other setup scripts if they're being run by easy_install, unless you use a try/finally block to remove your "tweak" afterwards. Subclassing is therefore the recommended (and documented) procedure. From pje at telecommunity.com Tue Jan 9 20:47:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 14:47:12 -0500 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked Message-ID: <5.1.1.6.0.20070109144659.04477400@sparrow.telecommunity.com> At 09:51 AM 1/9/2007 +0100, Ronald Oussoren wrote: >Another alternative is to tweak >distutils.commands.build.build.sub_commands, which is more convenient >than subclassing. But will crash other setup scripts if they're being run by easy_install, unless you use a try/finally block to remove your "tweak" afterwards. Subclassing is therefore the recommended (and documented) procedure. From pje at telecommunity.com Tue Jan 9 20:53:21 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 14:53:21 -0500 Subject: [Distutils] Adding a setuptool command that runs when setup.py build is invoked In-Reply-To: <5.1.1.6.0.20070109144659.04477400@sparrow.telecommunity.co m> Message-ID: <5.1.1.6.0.20070109145242.04475ce8@sparrow.telecommunity.com> At 02:47 PM 1/9/2007 -0500, Phillip J. Eby wrote: >At 09:51 AM 1/9/2007 +0100, Ronald Oussoren wrote: Sorry for the duplicate message, I was trying to see if there was a problem with the list, which has held my two setuptools release announcements From pje at telecommunity.com Tue Jan 9 19:38:18 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 13:38:18 -0500 Subject: [Distutils] setuptools 0.6c4 released Message-ID: <5.1.1.6.0.20070109131725.04214350@sparrow.telecommunity.com> As of this release, the installation instructions have been changed to support installation via RPM, .exe (Windows) or "sh setuptools-whatever.egg". In other words, ez_setup.py is no longer needed or recommended for user installation of setuptools. Prior to the final release of setuptools 0.6, I intend to update ez_setup to do one or both of the following things: 1. It will detect and download the current stable setuptools release from the Cheeseshop, AND/OR 2. It will instruct the user to visit the cheeseshop page and follow the installation instructions there. I'm not sure yet which is the best approach, so I'll probably just do both. :) I'd like to hear some input, but no "voting" please; just explain your rationale one way or the other, with detailed information about your situation/use cases. Thanks! From pje at telecommunity.com Tue Jan 9 20:35:26 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 14:35:26 -0500 Subject: [Distutils] setuptools 0.6c5 released :( Message-ID: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> I found some upload and bdist related issues while uploading 0.6c4, so I've fixed them and released 0.6c5. If you haven't seen the release announcement for 0.6c4 yet, it's because it has been held for moderation because of "a suspicious header". Why the release of setuptools 0.6c4 is considered suspicious, I don't know. Hopefully this message won't also be considered suspicious. :( From jnelson-distutils at securepipe.com Tue Jan 9 22:34:58 2007 From: jnelson-distutils at securepipe.com (Jon Nelson) Date: Tue, 09 Jan 2007 15:34:58 -0600 Subject: [Distutils] setuptools 0.6c5 released :( In-Reply-To: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> Message-ID: <45A40A82.1000102@securepipe.com> Phillip J. Eby wrote: > I found some upload and bdist related issues while uploading 0.6c4, so I've > fixed them and released 0.6c5. I found a bug in 0.6c5: setuptools/command/bdist_rpm.py is missing an import: from distutils.sysconfig import get_python_version get_python_version is used but never defined. From jnelson-distutils at securepipe.com Tue Jan 9 22:53:55 2007 From: jnelson-distutils at securepipe.com (Jon Nelson) Date: Tue, 09 Jan 2007 15:53:55 -0600 Subject: [Distutils] setuptools 0.6c5 released :( In-Reply-To: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> Message-ID: <45A40EF3.2010300@securepipe.com> Phillip J. Eby wrote: > I found some upload and bdist related issues while uploading 0.6c4, so I've > fixed them and released 0.6c5. Sorry, found another but in 0.6c5 - it won't build it's own binary rpm (bdist_rpm). :-( I only get a src.rpm. 0.6c3 builds both a .src.rpm and a .noarch.rpm. From pje at telecommunity.com Tue Jan 9 23:25:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 09 Jan 2007 17:25:53 -0500 Subject: [Distutils] setuptools 0.6c5 released :( In-Reply-To: <45A40A82.1000102@securepipe.com> References: <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070109172139.02782080@sparrow.telecommunity.com> At 03:34 PM 1/9/2007 -0600, Jon Nelson wrote: >Phillip J. Eby wrote: > > I found some upload and bdist related issues while uploading 0.6c4, so > I've > > fixed them and released 0.6c5. > >I found a bug in 0.6c5: > >setuptools/command/bdist_rpm.py is missing an import: > >from distutils.sysconfig import get_python_version > >get_python_version is used but never defined. Ugh. I fixed this one back in September on the trunk, but evidently missed the backport: http://svn.python.org/view/sandbox/trunk/setuptools/setuptools/command/bdist_rpm.py?rev=51967&r1=51960&r2=51967 I've copied it and am making preparations for 0.6c6. From davidf at sjsoft.com Wed Jan 10 14:32:46 2007 From: davidf at sjsoft.com (David Fraser) Date: Wed, 10 Jan 2007 15:32:46 +0200 Subject: [Distutils] setuptools 0.6c4 released In-Reply-To: <5.1.1.6.0.20070109131725.04214350@sparrow.telecommunity.com> References: <5.1.1.6.0.20070109131725.04214350@sparrow.telecommunity.com> Message-ID: <45A4EAFE.9010909@sjsoft.com> Phillip J. Eby wrote: > As of this release, the installation instructions have been changed to > support installation via RPM, .exe (Windows) or "sh > setuptools-whatever.egg". In other words, ez_setup.py is no longer needed > or recommended for user installation of setuptools. > Fantastic! That was my major remaining gripe with setuptools :-) And its great to see the installers all available on pypi (in 0.6c5 form now of course) > Prior to the final release of setuptools 0.6, I intend to update ez_setup > to do one or both of the following things: > > 1. It will detect and download the current stable setuptools release from > the Cheeseshop, AND/OR > > 2. It will instruct the user to visit the cheeseshop page and follow the > installation instructions there. > > I'm not sure yet which is the best approach, so I'll probably just do both. :) > What about 3. It will ask the user if it wants to do 1, and otherwise do 2. David From pje at telecommunity.com Wed Jan 10 15:18:38 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 10 Jan 2007 09:18:38 -0500 Subject: [Distutils] setuptools 0.6c4 released In-Reply-To: <45A4EAFE.9010909@sjsoft.com> References: <5.1.1.6.0.20070109131725.04214350@sparrow.telecommunity.com> <5.1.1.6.0.20070109131725.04214350@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070110091513.04830e88@sparrow.telecommunity.com> At 03:32 PM 1/10/2007 +0200, David Fraser wrote: >Phillip J. Eby wrote: >>Prior to the final release of setuptools 0.6, I intend to update ez_setup >>to do one or both of the following things: >> >>1. It will detect and download the current stable setuptools release from >>the Cheeseshop, AND/OR >> >>2. It will instruct the user to visit the cheeseshop page and follow the >>installation instructions there. >> >>I'm not sure yet which is the best approach, so I'll probably just do >>both. :) >> >What about >3. It will ask the user if it wants to do 1, and otherwise do 2. Um, no. :) First, you didn't supply a rationale or usage scenario to justify that, and second, setup scripts are never interactive. (Or at least, I've never seen one that is.) Remember, ez_setup isn't being run on its own any more (although it will still support that). We're talking about what will happen when you run "setup.py install" (or similar) on a project that uses setuptools, when setuptools is not installed. The current mechanism of delaying 15 seconds before starting the download is about as interactive as it gets. So, when I say, "do both", what I mean is that it would display instructions (#2) and delay 15 seconds or so before starting #1. From lutz_p at gmx.net Wed Jan 10 16:32:33 2007 From: lutz_p at gmx.net (=?iso-8859-1?Q?=22Lutz_P=E4like=22?=) Date: Wed, 10 Jan 2007 16:32:33 +0100 Subject: [Distutils] custom egg metadata Message-ID: <20070110153233.178970@gmx.net> > > there is no real need to have the package_dir directly alongside the setup.py ? > > I have only a single setup.py, but lots of subprojects that i want to > > keep in subfolders. And there are no setup.py in the project subfolders > > but everything is controlled through the single setup.py and a config > > file within each project. > That's not really a supported arrangement; note that you won't be able to > make proper source distributions, for example. (Source distributions must > contain a setup.py to be installable.) I am aware that this is not a standard setup, but nevertheless that's the way i want to go. All the distribution will happen in-house without source and never hit the real world so source distributions won't be a issue here. Thanks for pointing this out though. > To be blunt, the distutils doesn't really support what you're trying to > do. And setuptools inherits certain of its limitations, like the fact > that it is built to assume that there is a setup.py in the current > directory and the current directory is the root of a project. I really would like to have more flexibility for the distribution of packages and setuptools are a major improvement over standard distutils. Thanks for all your effort to improve the situation. This is highly appreciated. As i already mentioned i would like to have better api support to use some functionality isolated from the rest of the package. I can see that distutils dictates a lot of decisions here but some design decisions are not clear to me. What comes to my mind for example is get_svn_revision(). This is a method of the egg_info command but it would also be useful to have this as a standalone function. Personally i replicated it as a function to take a filesystem path as parameter to get the svn revision of the given path. Very useful functionality but not very accessible within setuptools ... wouldn't it nice and easy to convert the method call to a function call ? Probably there are more low hanging fruits but i still need to take a closer look to the code. christmas is over but if the wishlist is still open, i would like to see the core setuptools functionality split off and to have a distutils compatibility layer on top. It is great to have pkg_resources.py on the distribution side, but it would be nice to have the same flexibility on the building side ... Regards, Lutz Paelike -- Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! Ideal f?r Modem und ISDN: http://www.gmx.net/de/go/smartsurfer From arve.knudsen at gmail.com Mon Jan 15 14:16:36 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Mon, 15 Jan 2007 14:16:36 +0100 Subject: [Distutils] sdist command doesn't include data files Message-ID: Hi I'm experiencing a problem when using setuptools (0.6c5) to generate a source tarball, using the 'sdist' command. What happens is that data files (PNGs) are omitted even though I include them in package_data (keyword to setup), and they are correctly included when installing. Is it the intended behaviour of 'sdist' to ignore package_data, or is this a bug? Thanks, Arve Knudsen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070115/43ac5cdc/attachment.htm From pje at telecommunity.com Mon Jan 15 17:30:41 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 15 Jan 2007 11:30:41 -0500 Subject: [Distutils] sdist command doesn't include data files In-Reply-To: Message-ID: <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> At 02:16 PM 1/15/2007 +0100, Arve Knudsen wrote: >Hi >I'm experiencing a problem when using setuptools (0.6c5) to generate a >source tarball, using the 'sdist' command. What happens is that data files >(PNGs) are omitted even though I include them in package_data (keyword to >setup), and they are correctly included when installing. Is it the >intended behaviour of 'sdist' to ignore package_data, or is this a bug? They should be included in the sdist. Are you using CVS or Subversion, and if so, are the .png files under revision control? That information will help me narrow down what the problem is. From yo.gi at web.de Mon Jan 15 18:53:28 2007 From: yo.gi at web.de (Jochen Kupperschmidt) Date: Mon, 15 Jan 2007 18:53:28 +0100 Subject: [Distutils] setuptools: easy_install version option Message-ID: <45ABBF98.3090609@web.de> Hi, I was wondering why there is no --version/-V option available for easy_install to get the setuptools version currently installed (I would welcome that option). Is there any specific reason for this or is there an easier, "better" way to get that information? And by easier, I do not mean digging into the site-packages directory or starting a Python interactive shell and reading the `setuptools.__version__` attribute. Also, the --help-commands option, as advertised by easy_install, Common commands: (see '--help-commands' for more) is not recognized. This was experienced with setuptools 0.6c5 and Python 2.5 on Windows XP. Regards, Jochen Kupperschmidt From arve.knudsen at gmail.com Mon Jan 15 20:20:45 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Mon, 15 Jan 2007 20:20:45 +0100 Subject: [Distutils] sdist command doesn't include data files In-Reply-To: <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> References: <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> Message-ID: Hi Philip On 1/15/07, Phillip J. Eby wrote: > > At 02:16 PM 1/15/2007 +0100, Arve Knudsen wrote: > >Hi > >I'm experiencing a problem when using setuptools (0.6c5) to generate a > >source tarball, using the 'sdist' command. What happens is that data > files > >(PNGs) are omitted even though I include them in package_data (keyword to > >setup), and they are correctly included when installing. Is it the > >intended behaviour of 'sdist' to ignore package_data, or is this a bug? > > They should be included in the sdist. Are you using CVS or Subversion, > and > if so, are the .png files under revision control? That information will > help me narrow down what the problem is. > After having sent this mail, I did some more digging on Google and from what I understand sdist doesn't actually support the 'package_data' keyword, right? This should really be better pointed out in the documentation, as some of my colleagues also struggled with this before. So I tried 'include_package_data' instead, after having extended setuptools with support for the Mercurial VCS. This made sdist include also each and every version-controlled Python file however. Is this the intended behaviour? So in the end I resorted to MANIFEST.in, although I'd prefer to keep this within setup.py. If there's any interest I can provide the source for my Mercurial extension (simple as it may be). Thanks, Arve Knudsen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070115/bfea92a7/attachment.htm From pje at telecommunity.com Mon Jan 15 20:54:37 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 15 Jan 2007 14:54:37 -0500 Subject: [Distutils] sdist command doesn't include data files In-Reply-To: References: <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070115144744.02a263a8@sparrow.telecommunity.com> At 08:20 PM 1/15/2007 +0100, Arve Knudsen wrote: >So I tried 'include_package_data' instead, after having extended >setuptools with support for the Mercurial VCS. This made sdist include >also each and every version-controlled Python file however. Is this the >intended behaviour? Yes. Why would you *not* include them? >If there's any interest I can provide the source for my Mercurial >extension (simple as it may be). May I suggest that you offer it as a Cheeseshop package instead? Then anyone who wants to use it can just install it. (I'm assuming, of course, that you wrote the extension as a "setuptools.file_finders" entry point.) From arve.knudsen at gmail.com Mon Jan 15 23:01:52 2007 From: arve.knudsen at gmail.com (Arve Knudsen) Date: Mon, 15 Jan 2007 23:01:52 +0100 Subject: [Distutils] sdist command doesn't include data files In-Reply-To: <5.1.1.6.0.20070115144744.02a263a8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070115112916.028c1740@sparrow.telecommunity.com> <5.1.1.6.0.20070115144744.02a263a8@sparrow.telecommunity.com> Message-ID: On 1/15/07, Phillip J. Eby wrote: > > At 08:20 PM 1/15/2007 +0100, Arve Knudsen wrote: > >So I tried 'include_package_data' instead, after having extended > >setuptools with support for the Mercurial VCS. This made sdist include > >also each and every version-controlled Python file however. Is this the > >intended behaviour? > > Yes. Why would you *not* include them? Because I have defined which *Python* packages I wish to include, include_package_data trips all over this by including other Python packages. I thought it was to add packages' data files? For me it is quite useless since I can't (to my knowledge) control its choice of files, apart from specifying exclusion patterns iff exclude_package_data is indeed recognized by sdist? Regards, Arve Knudsen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070115/13f4ed2f/attachment.htm From ianb at colorstudy.com Tue Jan 16 15:59:14 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 16 Jan 2007 09:59:14 -0500 Subject: [Distutils] Question re: workingenv.py and cli.exe In-Reply-To: <45AC4591.9010102@theorganization.net> References: <45AC4591.9010102@theorganization.net> Message-ID: <45ACE842.5020009@colorstudy.com> Jos Yule wrote: > I didn't know where else to address this - is there a mailing list or > group which would be better suited to this email? The distutils-sig group (copied) might be good. Especially when I'm unsure of the answer like now ;) > First - thanks for workingenv! Very helpful. > > I've run into a bug when using it to install Pylons on a WinXP box, > using easy_install.exe. > > When doing a "easy_install Pylons", after creating a new workingenv and > activating it, it gets hung up on a missing cli.exe from " dir>/lib/python2.4/setuptools", which on my computer is simply a > "__init__.py" and "__init__.pyc" file. I've copied the cli.exe from > "/\lib\python2.4\setuptools-0.6c5-py2.4.egg\setuptools". > > Anyway, it still seems to work, this is more of an FYI. Hi Jos. I'm not really sure what cli.exe is, but then I don't really use Windows. Maybe someone here can explain more? Hrm... or maybe my monkeypatch of setuptools is breaking the attempt to find cli.exe... :( (I still really wish I didn't have to monkeypatch setuptools, but I do) -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Tue Jan 16 16:16:52 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 16 Jan 2007 10:16:52 -0500 Subject: [Distutils] Question re: workingenv.py and cli.exe In-Reply-To: <45ACE842.5020009@colorstudy.com> References: <45AC4591.9010102@theorganization.net> <45AC4591.9010102@theorganization.net> Message-ID: <5.1.1.6.0.20070116101403.044f2428@sparrow.telecommunity.com> At 09:59 AM 1/16/2007 -0500, Ian Bicking wrote: >Jos Yule wrote: > > I didn't know where else to address this - is there a mailing list or > > group which would be better suited to this email? > >The distutils-sig group (copied) might be good. Especially when I'm >unsure of the answer like now ;) > > > First - thanks for workingenv! Very helpful. > > > > I've run into a bug when using it to install Pylons on a WinXP box, > > using easy_install.exe. > > > > When doing a "easy_install Pylons", after creating a new workingenv and > > activating it, it gets hung up on a missing cli.exe from " > dir>/lib/python2.4/setuptools", which on my computer is simply a > > "__init__.py" and "__init__.pyc" file. I've copied the cli.exe from > > "/\lib\python2.4\setuptools-0.6c5-py2.4.egg\setuptools". > > > > Anyway, it still seems to work, this is more of an FYI. > >Hi Jos. I'm not really sure what cli.exe is, but then I don't really >use Windows. cli.exe and gui.exe are the files setuptools uses to generate command-line and GUI script wrappers on Windows. easy_install.exe, for example, is just a copy of cli.exe. The way these files work is that they figure out what name they were invoked under, and then they find the adjacent corresponding .py file (e.g. easy_install-script.py), and run it, using the information in the #! line. Anyway, setuptools needs to have the .exe files in order to do this. From pje at telecommunity.com Tue Jan 16 16:19:44 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 16 Jan 2007 10:19:44 -0500 Subject: [Distutils] setuptools 0.6c5 released :( In-Reply-To: <45A41FB5.4020604@securepipe.com> References: <5.1.1.6.0.20070109172139.02782080@sparrow.telecommunity.com> <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> <5.1.1.6.0.20070109172139.02782080@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070116101749.028856c8@sparrow.telecommunity.com> At 05:05 PM 1/9/2007 -0600, Jon Nelson wrote: >Say, I've got a local patch to setuptools that adds rpm 'Requires' >lines for all dependencies, which is a huge boon to us that use rpm for >everything (and rely on /it/ for dependency management). Interested? > >It's a pretty small patch, and I'm probably doing it wrong anyway, but >it works with everything I've thrown at it so far. The main problem I see is that there is no guaranteed mapping between RPM package names and PyPI package names. I'm also not clear on what scope you're trying to deal with the issue in. From jeremy.fix at gmail.com Wed Jan 17 10:36:09 2007 From: jeremy.fix at gmail.com (=?ISO-8859-1?Q?J=E9r=E9my_Fix?=) Date: Wed, 17 Jan 2007 10:36:09 +0100 Subject: [Distutils] Problem with libraries linking of intricated packages Message-ID: <375d6bb20701170136w2afe744fj4a481c91e192e6d3@mail.gmail.com> Hello ! I've got few questions about defining setup.py My project consits of intricated packages : setup.py /Pack1 --__init__.py -- sources_pack1 --/Pack2 -- -- __init__.py -- -- sources_pack2 The package pack2 uses librairies defined in Pack1; My setup.py looks like : pack1_srcs = glob.glob ("pack1/*.cc") pack1_ext = Extension ( 'Pack1', sources = pack1_srcs, libraries = ['boost_python'], include_dirs = [include_dir], extra_objects=[] ) pack2_srcs = glob.glob ("Pack1/Pack2/*.cc") pack2_ext = Extension ( 'Pack2', sources = Pack2_srcs, libraries = ['boost_python'], include_dirs = [include_dir], extra_objects=[] ) setup (name='package, version = '1.0', author = '***', author_email = '***', url = '***', description = "****", packages = ['Pack1, 'Pack2'], ext_modules = [pack1_ext, pack2_ext], data_files= [(glob.glob("pack1/*.h"), ("pack1/pack2/*.h"))] ) The compilation ./setup.py works well, but it seems that there is a problem with the linking of the librairies. When i launch a script using pack2, trying to import things from pack1, i've got undefined symbols (of symbols defined in pack1) On the other hand, if i succeed in compiling/installing pack1, then adding Pack1 to extra_objects of Pack2 makes it work ! Then i'm wondering if there is a mean to compile pack1, then telling to setup.py that it should use that library compiled before. Probably it is not possible and i should define two separated directories like setup.py Pack1/ -- __init__.py -- pack1_sources Pack2/ -- __init__py -- pack2_sources so that the first library is compiled and then , the compilation of the second one uses the first one ... i don't know. Thanks for your help ! Jeremy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070117/73e8f0fb/attachment.htm From pje at telecommunity.com Wed Jan 17 18:08:22 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 17 Jan 2007 12:08:22 -0500 Subject: [Distutils] setuptools 0.6c5 released :( In-Reply-To: <45AE2FFC.1060500@securepipe.com> References: <5.1.1.6.0.20070116101749.028856c8@sparrow.telecommunity.com> <5.1.1.6.0.20070109172139.02782080@sparrow.telecommunity.com> <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> <5.1.1.6.0.20070109143332.02863fe8@sparrow.telecommunity.com> <5.1.1.6.0.20070109172139.02782080@sparrow.telecommunity.com> <5.1.1.6.0.20070116101749.028856c8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070117115850.028a0430@sparrow.telecommunity.com> [Please stop taking this off-list; I don't do off-list support.] At 08:17 AM 1/17/2007 -0600, Jon Nelson wrote: >Phillip J. Eby wrote: > > At 05:05 PM 1/9/2007 -0600, Jon Nelson wrote: > >> Say, I've got a local patch to setuptools that adds rpm 'Requires' > >> lines for all dependencies, which is a huge boon to us that use rpm for > >> everything (and rely on /it/ for dependency management). Interested? > >> > >> It's a pretty small patch, and I'm probably doing it wrong anyway, but > >> it works with everything I've thrown at it so far. > > > > The main problem I see is that there is no guaranteed mapping between > > RPM package names and PyPI package names. I'm also not clear on what > > scope you're trying to deal with the issue in. > >Well, in theory, the change would only affect those using setuptools >*and* bdist_rpm. If using bdist_rpm, then it only /adds/ dependency >information. As far as I can tell, unless these requirements are prefixed with somethign to indicate their CheeseShop/setuptools nature, *and* a corresponding Provides: is done in the RPM for what *that* package provides, I don't really see how that would work. The difficulty I see in doing that, especially as a default setting, is that it means dependency RPMs might need to be rebuilt as setuptools RPMs. Essentially, I don't see how this can work as a default behavior, and I'd like some input from other people using RPMs before considering adding this as a normal behavior. Note too, by the way, that there is absolutely no need for you to patch setuptools in order to get this behavior. just create a project directory like this: setup.py my_bdist_rpm.py And define a subclass of setuptools' bdist_rpm in my_bdist_rpm.py, list it in your 'distutils.commands' entry points (see setuptools' own setup.py for examples), and then run "setup.py install". You will then have a new command installed on your machine, that you can run with *any* setuptools-based package, or by using this trick to run a non-setuptools' package's setup.py with your command: python -c "import setuptools;execfile('setup.py')" my_bdist_rpm ... So, there's nothing stopping you from having this behavior without needing to maintain a patch; just maintain a subclass instead. You can even distribute your extended command on the Cheeseshop, so that other people can install it with easy_install and be able to use the same behavior, if they prefer it. Some popularity might then be convincing with respect to including it in 0.7 as a default behavior. >I'll send you the patch anyway. What would really be ideal, IMO, would >be for it to default to on (again, it only changes bdist_rpm behavior) >and for it to be switchable somehow. For environments that use RPM >heavily (like all of the environments I've worked in for almost 10 years >now) it's a huge bonus. > > >diff -ur setuptools-0.6c5.orig/setuptools/command/bdist_rpm.py >setuptools-0.6c5/setuptools/command/bdist_rpm.py >--- setuptools-0.6c5.orig/setuptools/command/bdist_rpm.py >2007-01-09 13:20:24.000000000 -0600 >+++ setuptools-0.6c5/setuptools/command/bdist_rpm.py 2007-01-09 >16:59:01.000000000 -0600 >@@ -4,7 +4,9 @@ > # finally, a kludge to track .rpm files for uploading when run on >Python <2.5. > > from distutils.command.bdist_rpm import bdist_rpm as _bdist_rpm >-import sys, os >+from distutils.sysconfig import get_python_version >+import sys, os, re >+from pkg_resources import * > > class bdist_rpm(_bdist_rpm): > >@@ -12,6 +14,21 @@ > _bdist_rpm.initialize_options(self) > self.no_egg = None > >+ def finalize_options(self): >+ _bdist_rpm.finalize_options(self) >+ requires = list( yield_lines(self.distribution.install_requires >or ()) ) >+ if not requires: return >+ if not self.requires: >+ self.requires = [] >+ # dang. RPM requires versions in a different format (w/spaces!) >+ VERS_RE = re.compile(r"\s*(<=?|>=?|==|!=)\s*((\w|[-.])+)") >+ for req in requires: >+ parts = VERS_RE.split(req)[:3] >+ parts[0] = safe_name(parts[0]) >+ if len(parts) > 1: >+ parts[2] = safe_version(parts[2]) >+ self.requires.append(' '.join(parts)) >+ > if sys.version<"2.5": > # Track for uploading any .rpm file(s) moved to self.dist_dir > def move_file(self, src, dst, level=1): From constant.beta at gmail.com Thu Jan 18 14:50:51 2007 From: constant.beta at gmail.com (=?ISO-8859-2?Q?Micha=B3_Kwiatkowski?=) Date: Thu, 18 Jan 2007 14:50:51 +0100 Subject: [Distutils] How to get a list of package releases Message-ID: <5e8b0f6b0701180550t341939aewd9eadbab3d8f5ccb@mail.gmail.com> [CC-ing this to distutils-sig, because it relates to setuptools as well, but please keep discussion on the catalog-sig if possible.] Hi, Maybe this is a stupid question, but how can I list all versions of a package that were registered on PyPI? For example, when I go to http://python.org/pypi/setuptools/ I'm redirected to the page of last version of setuptools. How can I check earlier versions of setuptools from there? I know this functionality is available for project admins, what about users? Another thing, some packages offer a development versions by giving a link to their subversion repository. Is there a way to list these extra install options on PyPI? By "extra options" I mean all non-standard version strings that will work with `easy_install package==version`, like common "dev". Related setuptools question: is there a way to get list of available versions from easy_install? Cheers, mk From jpellerin at gmail.com Thu Jan 18 17:26:29 2007 From: jpellerin at gmail.com (jason pellerin) Date: Thu, 18 Jan 2007 11:26:29 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs Message-ID: <3bb02d620701180826v42930ea3sab3b016c4070c279@mail.gmail.com> Hello all. I'm the primary author of nose. Michal Kwiatkowski was kind enough to point me to this thread, and I wanted to chime in with a few questions and answers. Apologies for pulling things from various parts of the thread together, I didn't want to send multiple replies all covering the same ground. Nose works fine with setuptools in source packages. It provides nose.collector for use as a test_suite with the test command, and a nosetests command that uses nose's own testrunner and supports loading options from setup.cfg. So for today's use I think it works pretty well. That's the good news. The bad: PJE wrote: > (I'm not sure if nose really works with eggs, though; ScanningLoader will > discover tests in eggs as long as the source is included, but nose may be > strictly file-based for all I know. Changing it probably wouldn't be too > difficult, however.) There's no explicit egg support and, unlike ScanningLoader, nose uses naive os and os.path functions to access the filesystem, so I think I'll need to supply a 2nd loader than uses pkg_resources. > The loader must support a .loadTestFromName(name, obj) call, where name > will be an empty string, and obj will be the object specified by a test > entry point. The return value must be a unittest.TestSuite. And it must > be possible to call the loader multiple times with different 'obj' values. This is broken in nose 0.9.x -- I have the logic of loadTestsFromName backwards, in that it only uses the module argument to determine the package context of the name to be loaded; if the name is blank and the module is 'foo' it will try to load tests from 'foo.' Not so good. It also currently is a generator that yields tests as they are loaded, rather finding them all and returning a TestSuite -- I don't know whether that will also be a problem. And nose currently uses imp.find_module and imp.load_module to import test modules, since that seemed to be the easiest way to handle the very common case of multiple test modules with the same name in different (non-package) directories. I don't know whether those will work correctly when importing from an egg. I'm hoping to fix all of these loader problems in the 0.10 series. I'll make whatever changes will be needed to support loading from eggs and the final form of this proposal (so long as that doesn't break anything that works now), but as you can see I'm going to need some guidance about what I need to do. JP From pje at telecommunity.com Thu Jan 18 17:54:11 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 18 Jan 2007 11:54:11 -0500 Subject: [Distutils] [Catalog-sig] How to get a list of package releases In-Reply-To: <5e8b0f6b0701180550t341939aewd9eadbab3d8f5ccb@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070118115051.045cb688@sparrow.telecommunity.com> At 02:50 PM 1/18/2007 +0100, =?ISO-8859-2?Q?Micha=B3_Kwiatkowski?= wrote: >Related setuptools question: is there a way to get list of available >versions from easy_install? The setuptools.package_index.PackageIndex class knows about all discoverable versions. However, this doesn't extend to versions that have been hidden by the package author. To have more than one discoverable version at the PyPI level, the author has to have made them visible in the admin interface. But, whichever ones *are* visible can be discovered by the PackageIndex class, along with any that are linked from the Home Page or Download URLs, or are linked on the package's PyPI page. More precisely, PackageIndex doesn't track versions; it tracks available *distributions*, which may include source distributions, SVN checkouts, eggs, .exe's, etc. But each distribution object carries version info determined from its filename. From j.m.h.thomas at dl.ac.uk Thu Jan 18 18:29:33 2007 From: j.m.h.thomas at dl.ac.uk (Jens Thomas) Date: Thu, 18 Jan 2007 17:29:33 +0000 Subject: [Distutils] package_data issue (chopping off characters) Message-ID: <45AFAE7D.1090904@dl.ac.uk> Hello, I've just hit the problem that was mentioned in the thread with the above title (see: http://mail.python.org/pipermail/distutils-sig/2005-April/004453.html). However, the email trail seemed to have hit a dead end and it wasn't clear to me whether this had been flagged as a bug or whether it was a problem with the way that setup.py was being used. I've included a summary of the problem as I've encountered it below, but would appreciate any clarification on whether this is a bug and I need to wait for patched version, or whether it's a problem with the way I'm using setup.py and there is a change/work around I can use. Many thanks, Jens (this is being run on Ubuntu Dapper:Python 2.4.3 (#2, Oct 6 2006, 07:52:30) [GCC 4.0.3 (Ubuntu 4.0.3-1ubuntu5)] on linux2 ) I have the following directory tree dir1/ setup.py __init__.py dir2/ __init__.py doc/ test.txt And the following setup.py: #!/usr/bin/env python from distutils.core import setup setup( packages=['dir1','dir1.dir2'], package_dir = {'dir1': ''}, package_data={'dir1': ['doc/*.txt']} ) I need this done in this way because of the way that my CVS repository has been set up, with the root directory of the source (where the setup.py script lives) also being the root directory of the package, and I thought that the package_dir line above took care of that. On running the script, however, I get: jmht at csevig6:~/dir1$ python setup.py install running install running build running build_py creating build creating build/lib creating build/lib/dir1 copying __init__.py -> build/lib/dir1 creating build/lib/dir1/dir2 copying dir2/__init__.py -> build/lib/dir1/dir2 creating build/lib/dir1/oc error: can't copy 'oc/test.txt': doesn't exist or not a regular file This appears to be caused by a bit in the function get_data_files in the file: /usr/lib/python2.4/distutils/command/build_py.py specifically the line: plen = len(src_dir)+1 If len(src_dir) is 0, which it is in my case, the first character of the directory gets chopped off. From jim at zope.com Mon Jan 22 15:33:11 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 22 Jan 2007 09:33:11 -0500 Subject: [Distutils] package_data not used by sdist command Message-ID: <45B4CB27.3000005@zope.com> It appears that the sdist command doesn't consult the data passed in the package_data option to setup. I want to name data that isn't under version control for inclusion. Because sdist doesn't pay attention to package_data, the data I'm naming isn't included in the source distribution. It is included if I build an egg directly. This appears to me to be a bug. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 22 16:14:57 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 22 Jan 2007 10:14:57 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> Message-ID: <45B4D4F1.6040401@zope.com> Phillip J. Eby wrote: > At 01:11 PM 1/5/2007 -0500, Jim Fulton wrote: >> Phillip J. Eby wrote: >> ... >>>> Why can't an entry point invoke a test loader itself? >>>> This seems much simpler and more straightforward to me. >>> Because that requires you to write code for something that can >>> adequately be expressed through an existing configuration mechanism. >>> *And* you have to write that code in every project. >> >> It's probably a couple of lines. Import a loader and call it. >> That doesn't seem like a big deal to me. Nice and explicit too. > > And it's enormously repetitive if you have a lot of projects. > > >> Besides, I expect most projects will have a single test suite >> (function) that could just be named directly. > > You expect wrongly. :) I can only speak based on my own experience. Projects like Zope and ZODB certainly want some sort of tool to find and assemble tests. Eggs are allowing and encouraging us to move away from such large projects. As we break up Zope into individual packages, I'm finding that the individual packages have single test suites. > 'nose' and 'py.test' are popular *precisely > because* they do NOT require this. Setuptools changed to emulate them > by adding ScanningLoader, even though it originally did the "test suite > function" thing you're proposing. > > See, I originally wrote all my test suites the way you're proposing, but > I changed because this is one of those "convention beats configuration" > situations. It's easier to just conform to a policy and have a tool > that applies the policy. Different projects may have different > policies, so providing an option for the loader, allows them to follow > it. (And it made my life easier, too, as I began doing more small > projects instead of a few large ones.) > > And, if you want your approach to be widely adopted, it would be best to > allow projects to follow their own policies without requiring them to > add any code, even if it's a one-liner. They'll be adding code one way or another. Either they add it to their test modules, or they add it to setup.py. The advantage of the former is that it provides more flexibility without requiring any more work. It is also more explicit. > So, *technically*, there may be no reason to do this, but from a > usability, friendliness, compatiblity, marketing, adoption, etc. format > I don't think the standard will be successful without allowing this > out. People often have heavy psychological investment in existing > tools, and it is hundreds of times easier to rationalize adding a line > of configuration to setup() than it is to rationalize adding two or > three lines of code. One is merely a packaging change, the other is a > programming change. We disagree. I don't have any problem with having tools that can be used to find and assemble tests into a test suite. I find the unittest loader framework to be baroque. I'd rather let people use whatever framework they want to do this and to control this from Python. I withdraw the proposal. I wouldn't object to the proposal you want, if you want to write it. I don't have the time or interest to write it myself. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jan 22 16:35:17 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 22 Jan 2007 10:35:17 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <45B4D4F1.6040401@zope.com> References: <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070122102821.0444ca10@sparrow.telecommunity.com> At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote: >I'd rather let people use whatever framework >they want to do this and to control this from Python. The approach I suggested certainly allows that to happen, and if the default loader is either unittest's default loader or setuptools' scanning loader, then it works for Zope's use cases without any change from the way you'd like it. I just want it to *also* be possible for people to specify a non-default suite loader. People who are using non-default loaders now, already have them specified in their setup.py. If this were part of the new spec, they would need only to change (or duplicate) the test_loader and test_suite to be entry points instead of setup() keywords, and they would then work with the test declaration system. This is a much lower barrier to entry in both work needed to switch over, and (IMO) in psychological objection to doing so. From jim at zope.com Mon Jan 22 16:59:36 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 22 Jan 2007 10:59:36 -0500 Subject: [Distutils] RFC: Standard Declaration of tests in eggs In-Reply-To: <5.1.1.6.0.20070122102821.0444ca10@sparrow.telecommunity.com> References: <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105115117.04ce1f78@sparrow.telecommunity.com> <5.1.1.6.0.20070105122701.04265ec0@sparrow.telecommunity.com> <5.1.1.6.0.20070105132114.041c8dd8@sparrow.telecommunity.com> <5.1.1.6.0.20070122102821.0444ca10@sparrow.telecommunity.com> Message-ID: <45B4DF68.2080902@zope.com> Phillip J. Eby wrote: > At 10:14 AM 1/22/2007 -0500, Jim Fulton wrote: >> I'd rather let people use whatever framework >> they want to do this and to control this from Python. > > The approach I suggested certainly allows that to happen, I know. That's why I wouldn't object to it. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From anthony at interlink.com.au Wed Jan 24 07:05:27 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 24 Jan 2007 17:05:27 +1100 Subject: [Distutils] setuptools requires bash, not sh. Message-ID: <200701241705.28143.anthony@interlink.com.au> From the top of an egg file: #!/bin/sh if [[ `basename $0` = "setuptools-0.6c5-py2.4.egg" ]] then ... That syntax is a bashism, not /bin/sh. -bash-2.05b$ sh ./setuptools-0.6c5-py2.4.egg --prefix=~ ./setuptools-0.6c5-py2.4.egg: [[: not found ./setuptools-0.6c5-py2.4.egg is not the correct name for this egg file. Please rename it back to setuptools-0.6c5-py2.4.egg and try again. This is broken for Solaris (which doesn't have bash, by default), on recent Ubuntu (which has /bin/dash as /bin/sh, not bash) and any other Unix that doesn't have bash as /bin/sh. -- Anthony Baxter It's never too late to have a happy childhood. From skip at pobox.com Wed Jan 24 13:11:57 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 24 Jan 2007 06:11:57 -0600 Subject: [Distutils] Myghty.org down borked easy_install pylons Message-ID: <17847.19725.199197.19665@montanaro.dyndns.org> Originally posted to comp.lang.python... skip> I'm trying to easy_install pylons. Alas: skip> $ easy_install pylons skip> Searching for pylons skip> Best match: Pylons 0.9.4.1 skip> Processing Pylons-0.9.4.1-py2.5.egg skip> Pylons 0.9.4.1 is already the active version in easy-install.pth skip> Using /usr/local/mojam/lib/python2.5/site-packages/Pylons-0.9.4.1-py2.5.egg skip> Processing dependencies for pylons skip> Searching for Myghty>=1.1 skip> Reading http://cheeseshop.python.org/pypi/Myghty/ skip> Reading http://cheeseshop.python.org/pypi/Myghty/1.1 skip> Reading http://www.myghty.org skip> error: Download error: (111, 'Connection refused') skip> Is it not able to get past this point because there is only one skip> source for Myghty or does it always need to go to a package's home skip> page? I was able to work around this by manually downloading the skip> Myghty-1.1.tar.gz file and installing it. to which Robert Kern responded: Robert> Asking on Distutils-SIG list will be more likely to get Phillip Robert> Eby's attention. It surprises me that setuptools didn't pick up Robert> the tarball on the CheeseShop. then later: Robert> Now that www.myghty.org is back up, I see that the CheeseShop Robert> tarball is picked up eventually. I believe that easy_install Robert> checks all of the various sources to try to find a best match Robert> (for some definition of "best"). Of course, I would consider it Robert> a bug that an exception raised while trying to fetch one Robert> possible source stops the entire process, particularly when a Robert> matching tarball has already been found. So, Phillip, any thoughts on this? Skip From pje at telecommunity.com Wed Jan 24 16:25:30 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 24 Jan 2007 10:25:30 -0500 Subject: [Distutils] setuptools requires bash, not sh. In-Reply-To: <200701241705.28143.anthony@interlink.com.au> Message-ID: <5.1.1.6.0.20070124101057.04d7f890@sparrow.telecommunity.com> At 05:05 PM 1/24/2007 +1100, Anthony Baxter wrote: > From the top of an egg file: > >#!/bin/sh >if [[ `basename $0` = "setuptools-0.6c5-py2.4.egg" ]] >then ... > >That syntax is a bashism, not /bin/sh. I've changed it to use single '[ ]' instead of doubled ones. Is that the right syntax for non-bash shells? From pje at telecommunity.com Wed Jan 24 16:43:57 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 24 Jan 2007 10:43:57 -0500 Subject: [Distutils] Myghty.org down borked easy_install pylons In-Reply-To: <17847.19725.199197.19665@montanaro.dyndns.org> Message-ID: <5.1.1.6.0.20070124104009.02989d50@sparrow.telecommunity.com> At 06:11 AM 1/24/2007 -0600, skip at pobox.com wrote: > Robert> Now that www.myghty.org is back up, I see that the CheeseShop > Robert> tarball is picked up eventually. I believe that easy_install > Robert> checks all of the various sources to try to find a best match > Robert> (for some definition of "best"). Of course, I would consider it > Robert> a bug that an exception raised while trying to fetch one > Robert> possible source stops the entire process, particularly when a > Robert> matching tarball has already been found. > >So, Phillip, any thoughts on this? I've changed this in SVN to issue a warning instead, if the page being retrieved is merely a source of links, rather than an actual download. You can now update to the fixed version with "easy_install setuptools==dev" or "easy_install setuptools==dev06". From pje at telecommunity.com Wed Jan 24 16:50:49 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 24 Jan 2007 10:50:49 -0500 Subject: [Distutils] Path With Space Bug in EasyInstall In-Reply-To: <5BF747495606214DB06BE373D5313E77203D5A@kong.SHFF.local> Message-ID: <5.1.1.6.0.20070124104757.05136228@sparrow.telecommunity.com> At 11:01 AM 1/18/2007 -0600, Jon-Eric Simmons wrote: >I apologize if emailing you directly isn t the correct route to solve this >issue. The website isn t clear at all about how to report bugs. (Please >feel free to point me in the right direction.) Please direct all email regarding setuptools to the distutils-sig list. >In Windows XP, when I install a fresh version of Python in C:\Program >Files\Python2X (I tried both v2.4 & v2.5) and run ez_setup 0.6c5, when I >attempt to run easy_install I get some message like C:\Program is not a >valid executable. I need to know the exact message; please include the full text of what you did, both what you typed and what the program displayed. >However, if I perform the same steps with ez_setup 0.6c3, easy_install >succeeds. > >The release notes for 0.6c4 make it sound like an issue similar to this is >fixed, yet it appears to be broken by that version. Yes, and when I changed it to the way that works for you, it broke for someone else, so there is apparently more here than meets the eye. Please include the exact version of Windows that you're using, and whether you are using any Windows shell replacements (e.g. 4DOS/4NT, etc.). Thanks. From jsimmons at shff.com Wed Jan 24 17:29:23 2007 From: jsimmons at shff.com (Jon-Eric Simmons) Date: Wed, 24 Jan 2007 10:29:23 -0600 Subject: [Distutils] Path With Space Bug in EasyInstall In-Reply-To: <5.1.1.6.0.20070124104757.05136228@sparrow.telecommunity.com> References: <5BF747495606214DB06BE373D5313E77203D5A@kong.SHFF.local> <5.1.1.6.0.20070124104757.05136228@sparrow.telecommunity.com> Message-ID: <5BF747495606214DB06BE373D5313E77203EE5@kong.SHFF.local> I'm using WindowsXP SP2 and the included shell (no replacements). Repro steps: 1. Install Python 2.5 using python-2.5.msi. 2. Instead of the default installation path, install Python in 'C:\Program Files\Python25\'. 3. Add two directories to PATH: 'C:\Program Files\Python25' & 'C:\Program Files\Python25\Scripts' 4. Execute ez_setup.py v0.6c5. (See below for output.) 5. Execute easy_install to see if it worked. (See below for output.) ---- Begin ez_setup Output ---- Downloading http://cheeseshop.python.org/packages/2.5/s/setuptools/setuptools-0.6c5- py2.5.egg Processing setuptools-0.6c5-py2.5.egg Copying setuptools-0.6c5-py2.5.egg to c:\program files\python25\lib\site-packages Adding setuptools 0.6c5 to easy-install.pth file Installing easy_install-script.py script to C:\Program Files\Python25\Scripts Installing easy_install.exe script to C:\Program Files\Python25\Scripts Installing easy_install-2.5-script.py script to C:\Program Files\Python25\Scripts Installing easy_install-2.5.exe script to C:\Program Files\Python25\Scripts Installed c:\program files\python25\lib\site-packages\setuptools-0.6c5-py2.5.egg Processing dependencies for setuptools==0.6c5 ---- End ez_setup Output ---- ---- Begin easy_install Output ----- Cannot find Python executable C:\Program ---- End easy_install Output ----- If I do this with ez_setup v0.6c3, easy_install works just fine. Thanks, -Jon-Eric -----Original Message----- From: Phillip J. Eby [mailto:pje at telecommunity.com] Sent: Wednesday, January 24, 2007 9:51 AM To: Jon-Eric Simmons Cc: distutils-sig at python.org Subject: Re: Path With Space Bug in EasyInstall At 11:01 AM 1/18/2007 -0600, Jon-Eric Simmons wrote: >I apologize if emailing you directly isn t the correct route to solve this >issue. The website isn t clear at all about how to report bugs. (Please >feel free to point me in the right direction.) Please direct all email regarding setuptools to the distutils-sig list. >In Windows XP, when I install a fresh version of Python in C:\Program >Files\Python2X (I tried both v2.4 & v2.5) and run ez_setup 0.6c5, when I >attempt to run easy_install I get some message like C:\Program is not a >valid executable. I need to know the exact message; please include the full text of what you did, both what you typed and what the program displayed. >However, if I perform the same steps with ez_setup 0.6c3, easy_install >succeeds. > >The release notes for 0.6c4 make it sound like an issue similar to this is >fixed, yet it appears to be broken by that version. Yes, and when I changed it to the way that works for you, it broke for someone else, so there is apparently more here than meets the eye. Please include the exact version of Windows that you're using, and whether you are using any Windows shell replacements (e.g. 4DOS/4NT, etc.). Thanks. From pobrien at orbtech.com Wed Jan 24 18:34:45 2007 From: pobrien at orbtech.com (Patrick K. O'Brien) Date: Wed, 24 Jan 2007 11:34:45 -0600 Subject: [Distutils] Unknown hostname 'svn.eby-sarna.com' Message-ID: <45B798B5.9010500@orbtech.com> Phillip, Any idea what's happening with svn.eby-sarna.com? Best match: Pylons dev Downloading http://pylonshq.com/svn/Pylons/trunk#egg=Pylons-dev Doing subversion checkout from http://pylonshq.com/svn/Pylons/trunk to /tmp/easy_install-8X22CP/trunk subversion/libsvn_ra_svn/client.c:125: (apr_err=670007) svn: Unknown hostname 'svn.eby-sarna.com' Processing trunk Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-8X22CP/trunk/egg-dist-tmp-rrR2sh Traceback (most recent call last): -- Patrick K. O'Brien Orbtech http://www.orbtech.com Schevo http://www.schevo.org Louie http://www.pylouie.org From pje at telecommunity.com Wed Jan 24 21:04:10 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 24 Jan 2007 15:04:10 -0500 Subject: [Distutils] Unknown hostname 'svn.eby-sarna.com' In-Reply-To: <45B798B5.9010500@orbtech.com> Message-ID: <5.1.1.6.0.20070124150340.063185c0@sparrow.telecommunity.com> At 11:34 AM 1/24/2007 -0600, Patrick K. O'Brien wrote: >Any idea what's happening with svn.eby-sarna.com? The domain apparently expired; it's been renewed now. From pje at telecommunity.com Wed Jan 24 22:03:54 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 24 Jan 2007 16:03:54 -0500 Subject: [Distutils] Path With Space Bug in EasyInstall In-Reply-To: <5BF747495606214DB06BE373D5313E77203EE5@kong.SHFF.local> References: <5.1.1.6.0.20070124104757.05136228@sparrow.telecommunity.com> <5BF747495606214DB06BE373D5313E77203D5A@kong.SHFF.local> <5.1.1.6.0.20070124104757.05136228@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070124160200.0581f6d0@sparrow.telecommunity.com> At 10:29 AM 1/24/2007 -0600, Jon-Eric Simmons wrote: >---- Begin easy_install Output ----- >Cannot find Python executable C:\Program >---- End easy_install Output ----- Okay, I found the problem; it was a #!-parsing regression that happened in 0.6c4 when I was adding support for bdist_wininst installation of .exe wrappers. It's fixed now in SVN and will of course be out in 0.6c6. From schwehr at gmail.com Fri Jan 26 17:11:13 2007 From: schwehr at gmail.com (Kurt Schwehr) Date: Fri, 26 Jan 2007 11:11:13 -0500 Subject: [Distutils] failing to build quadtree if the package is already installed Message-ID: <37de72610701260811w7b7d1e1cp42ac7e737c32b07f@mail.gmail.com> Hi All, Sorry if this kind of thing has already been discussed. I am packaging quadtree for Mac OSX/fink (deb based package management) and am getting a failed build if the package is alread installed and I am trying to do a rebuild. Can someone point me at what needs to be changed? Is this because quadtree is a namespace? Thanks! -kurt cd externals/Quadtree && /sw/bin/python2.5 setup.py install --root=/sw/src/fink.build/root-quadtree-py25-0.1.1-1--single-version-externally-managed Traceback (most recent call last): File "setup.py", line 2, in from setuptools import setup, Extension File "/sw/lib/python2.5/site-packages/setuptools/__init__.py", line 2, in from setuptools.extension import Extension, Library File "/sw/lib/python2.5/site-packages/setuptools/extension.py", line 2, in from dist import _get_unpatched File "/sw/lib/python2.5/site-packages/setuptools/dist.py", line 6, in from setuptools.command.sdist import sdist File "/sw/lib/python2.5/site-packages/setuptools/command/sdist.py", line 3, in import os, re, sys, pkg_resources File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 2499, in add_activation_listener(lambda dist: dist.activate()) File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 599, in subscribe callback(dist) File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 2499, in add_activation_listener(lambda dist: dist.activate()) File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 2048, in activate map(declare_namespace, self._get_metadata('namespace_packages.txt')) File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 1667, in declare_namespace _handle_ns(packageName, path_item) File "/sw/lib/python2.5/site-packages/pkg_resources.py", line 1638, in _handle_ns loader.load_module(packageName); module.__path__ = path File "/sw/lib/python2.5/pkgutil.py", line 238, in load_module mod = imp.load_module(fullname, self.file, self.filename, self.etc) File "/sw/src/fink.build/quadtree-py25-0.1.1-1/pcl-0.11-20070118/externals/Quadtree/quadtree/__init__.py", line 2, in from _tree import Quadtree ImportError: No module named _tree cat quadtree/__init__.py from _tree import Quadtree And, here is the setup.py file: from setuptools import setup, Extension _tree = Extension('quadtree._tree', sources=['quadtree/_treemodule.c', 'shapelib/shptree.c', 'shapelib/shpopen.c'], include_dirs=['shapelib'] ) setup(name = 'Quadtree', version = '0.1.1', description = 'Quadtree spatial index for Python GIS', license = 'BSD', keywords = 'spatial index', author = 'Sean Gillies', author_email = 'sgillies at frii.com', maintainer = 'Sean Gillies', maintainer_email = 'sgillies at frii.com', url = 'http://icon.stoa.org/trac/pleiades/wiki/QuadTree ', packages = ['quadtree'], namespace_packages = ['quadtree'], ext_modules = [_tree], classifiers = [ 'Development Status :: 3 - Alpha', 'Intended Audience :: Developers', 'Intended Audience :: Science/Research', 'License :: OSI Approved :: BSD License', 'Operating System :: OS Independent', 'Programming Language :: C', 'Programming Language :: Python', 'Topic :: Scientific/Engineering :: GIS', 'Topic :: Database', ], ) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070126/2d8536d6/attachment.html From pje at telecommunity.com Fri Jan 26 18:03:07 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 26 Jan 2007 12:03:07 -0500 Subject: [Distutils] failing to build quadtree if the package is already installed In-Reply-To: <37de72610701260811w7b7d1e1cp42ac7e737c32b07f@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070126115632.0521dea0@sparrow.telecommunity.com> At 11:11 AM 1/26/2007 -0500, Kurt Schwehr wrote: >Sorry if this kind of thing has already been discussed. I am packaging >quadtree for Mac OSX/fink (deb based package management) and am getting a >failed build if the package is alread installed and I am trying to do a >rebuild. Can someone point me at what needs to be changed? Is this >because quadtree is a namespace? Yes, and it shouldn't be, since its __init__.py contains actual code, and is missing a namespace declaration. (The whole point of a namespace package is that the package itself shouldn't contain any code. After all, in a --single-version-externally-managed install, the __init__.py will not even be *there*.) So, this is a packaging error on the part of the quadtree developer(s). Most likely, quadtree is not really a namespace package and should not be declared as one, because the only reason to do that is if there are modules or subpackages under that package that are distributed *separately*. My guess is that the author saw a namespace declaration in some other project's setup.py and thought it was required boilerplate instead of an option. >20070118/externals/Quadtree/quadtree/__init__.py", line 2, in > from _tree import Quadtree >ImportError: No module named _tree > >cat quadtree/__init__.py > >from _tree import Quadtree > > >And, here is the setup.py file: > >from setuptools import setup, Extension >_tree = Extension('quadtree._tree', > sources=['quadtree/_treemodule.c', > 'shapelib/shptree.c', > 'shapelib/shpopen.c'], > include_dirs=['shapelib'] > ) >setup(name = 'Quadtree', > version = '0.1.1', > description = 'Quadtree spatial index for Python GIS', > license = 'BSD', > keywords = 'spatial index', > author = 'Sean Gillies', > author_email = ' sgillies at frii.com', > maintainer = 'Sean Gillies', > maintainer_email = 'sgillies at frii.com', > url = ' > http://icon.stoa.org/trac/pleiades/wiki/QuadTree', > packages = ['quadtree'], > namespace_packages = ['quadtree'], > ext_modules = [_tree], > classifiers = [ > 'Development Status :: 3 - Alpha', > 'Intended Audience :: Developers', > 'Intended Audience :: Science/Research', > 'License :: OSI Approved :: BSD License', > 'Operating System :: OS Independent', > 'Programming Language :: C', > 'Programming Language :: Python', > 'Topic :: Scientific/Engineering :: GIS', > 'Topic :: Database', > ], >) From schwehr at gmail.com Sun Jan 28 13:01:18 2007 From: schwehr at gmail.com (Kurt Schwehr) Date: Sun, 28 Jan 2007 07:01:18 -0500 Subject: [Distutils] failing to build quadtree if the package is already installed Message-ID: <37de72610701280401g2f06ecfch5d33f110bb932c1f@mail.gmail.com> Thanks! The package developer has fixed it and it is now up on the cheeseshop as version 0.1.2 of quadtree. -kurt > > Message: 2 > Date: Fri, 26 Jan 2007 12:03:07 -0500 > From: "Phillip J. Eby" > Subject: Re: [Distutils] failing to build quadtree if the package is > already installed > To: "Kurt Schwehr" , distutils-sig at python.org > Message-ID: > <5.1.1.6.0.20070126115632.0521dea0 at sparrow.telecommunity.com> > Content-Type: text/plain; charset="us-ascii"; format=flowed > > At 11:11 AM 1/26/2007 -0500, Kurt Schwehr wrote: > >Sorry if this kind of thing has already been discussed. I am packaging > >quadtree for Mac OSX/fink (deb based package management) and am getting a > >failed build if the package is alread installed and I am trying to do a > >rebuild. Can someone point me at what needs to be changed? Is this > >because quadtree is a namespace? > > Yes, and it shouldn't be, since its __init__.py contains actual code, and > is missing a namespace declaration. (The whole point of a namespace > package is that the package itself shouldn't contain any code. After all, > in a --single-version-externally-managed install, the __init__.py will not > even be *there*.) > > So, this is a packaging error on the part of the quadtree > developer(s). Most likely, quadtree is not really a namespace package and > should not be declared as one, because the only reason to do that is if > there are modules or subpackages under that package that are distributed > *separately*. My guess is that the author saw a namespace declaration in > some other project's setup.py and thought it was required boilerplate > instead of an option. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/distutils-sig/attachments/20070128/88e1913b/attachment.html From optilude at gmx.net Sun Jan 28 23:01:28 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 28 Jan 2007 22:01:28 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 12:21 PM 11/19/2006 -0500, Nathan R. Yergler wrote: >> I'm using zc.buildout for deploying a handful of applications, and one >> of our dependencies doesn't have eggs or "correctly" formatted pages >> (that would allow easy_install to track down the repository for direct >> building from source). Does zc.buildout provide any way to build an egg >>from a Subversion URL? Something with similar behavior to: >> easy_install -m -d ./eggs http://path/to/svn/repos >> >> I'm happy to write a recipe if necessary, just thought I'd make sure >> someone else hadn't already scratched this itch. > > If it's a dependency declared in your setup.py, you can do something like this: > > dependency_links=['http://path/to/svn/repos#egg=Foo-dev'], > install_requires=['Foo>=1.2,==dev'], > > in your setup() call, to depend on version 1.2 or greater of Foo; the ==dev > and the #egg=Foo-dev will allow it to be found via SVN. > > See also these two manual sections that document the use of ==dev and #egg=: > > http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion > > http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall I found myself searching for this very thing, and couldn't find a way to do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL in 'find-links' didn't work either. I'm not sure I understand zc.recipe.egg and setuptools well enough to be able to guess what may be wrong. Since zc.buildout uses setuptools though, if there is a standard setuptools way of doing this, I'd hope it should be possible to do it in zc.buildout? Martin From ianb at colorstudy.com Sun Jan 28 23:25:06 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 28 Jan 2007 16:25:06 -0600 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> Message-ID: <45BD22C2.7030403@colorstudy.com> Martin Aspeli wrote: >> http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion >> >> http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall > > I found myself searching for this very thing, and couldn't find a way to > do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a > syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL > in 'find-links' didn't work either. > > I'm not sure I understand zc.recipe.egg and setuptools well enough to be > able to guess what may be wrong. Since zc.buildout uses setuptools > though, if there is a standard setuptools way of doing this, I'd hope it > should be possible to do it in zc.buildout? Is there a way to do find-links in buildout? I think so, maybe named something like find-links. Then you'd point at a page that has the link with #egg=.... IIRC you have to point find-links at a real HTML page. I sometimes put on in svn (maybe actually in the buildout), and point it there. Annoying part is you have to commit it to test. buildout, kind of like install_requires, I think needs you to give it something it can actually fulfill. So you'd need Foo==dev,>1.1 or somesuch. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From pje at telecommunity.com Sun Jan 28 23:27:15 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 28 Jan 2007 17:27:15 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> At 10:01 PM 1/28/2007 +0000, Martin Aspeli wrote: >I found myself searching for this very thing, and couldn't find a way to >do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a >syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL >in 'find-links' didn't work either. Hm. easy_install accepts svn: URLs in --find-links as long as they have the #egg part, so that sounds like a problem with zc.buildout. From optilude at gmx.net Sun Jan 28 23:33:44 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 28 Jan 2007 22:33:44 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <45BD22C2.7030403@colorstudy.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> Message-ID: Ian Bicking wrote: > Is there a way to do find-links in buildout? I think so, maybe named > something like find-links. Yep. > Then you'd point at a page that has the link > with #egg=.... IIRC you have to point find-links at a real HTML page. Right. But I don't have such a page. Basically, I want my buildout to track e.g.: http://svn.plone.org/svn/plone/plone.portlets/trunk This having a setup.py and I hope other necessary info to check it out and build it. What would I need in terms of find-links for this to work? > I sometimes put on in svn (maybe actually in the buildout), and point it > there. Annoying part is you have to commit it to test. buildout, kind > of like install_requires, I think needs you to give it something it can > actually fulfill. So you'd need Foo==dev,>1.1 or somesuch. You mean in buildout's list of eggs-to-install? Basically, my buildout.cfg now has: find-links = http://download.zope.org/distribution/ http://effbot.org/downloads eggs = elementtree python-yadis python-openid python-urljr I believe I can do foo==dev or whatever in that list of eggs as well. But I'm not sure how that relates to svn link above. Martin From optilude at gmx.net Sun Jan 28 23:34:59 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 28 Jan 2007 22:34:59 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> References: <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 10:01 PM 1/28/2007 +0000, Martin Aspeli wrote: >> I found myself searching for this very thing, and couldn't find a way to >> do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a >> syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL >> in 'find-links' didn't work either. > > Hm. easy_install accepts svn: URLs in --find-links as long as they have > the #egg part, so that sounds like a problem with zc.buildout. Or a problem between the chair and the keyboard. :) To my knowledge, zc.buildout does use a custom version of easy_install.py though. I'm sure Jim can explain more. I will try a few more permutations to see if I can figure it out. Martin From pje at telecommunity.com Sun Jan 28 23:43:18 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 28 Jan 2007 17:43:18 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <45BD22C2.7030403@colorstudy.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: >Is there a way to do find-links in buildout? I think so, maybe named >something like find-links. Then you'd point at a page that has the link >with #egg=.... IIRC you have to point find-links at a real HTML page. Not for easy_install, you don't. If buildout doesn't support it, that's a bug or limitation of buildout. From ianb at colorstudy.com Sun Jan 28 23:44:16 2007 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 28 Jan 2007 16:44:16 -0600 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> References: <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> Message-ID: <45BD2740.5060206@colorstudy.com> Phillip J. Eby wrote: > At 10:01 PM 1/28/2007 +0000, Martin Aspeli wrote: >> I found myself searching for this very thing, and couldn't find a way to >> do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a >> syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL >> in 'find-links' didn't work either. > > Hm. easy_install accepts svn: URLs in --find-links as long as they have > the #egg part, so that sounds like a problem with zc.buildout. I don't know if it's a buildout thing. I was only doing it indirectly, and you can't point to something that isn't a URL or HTML page with --find-links AFAICT. I thought you couldn't point directly to a package with find-links (hence dependency_links), but I think I'm just confusing setup.py with easy_install. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From optilude at gmx.net Sun Jan 28 23:52:31 2007 From: optilude at gmx.net (Martin Aspeli) Date: Sun, 28 Jan 2007 22:52:31 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: >> Is there a way to do find-links in buildout? I think so, maybe named >> something like find-links. Then you'd point at a page that has the link >> with #egg=.... IIRC you have to point find-links at a real HTML page. > > Not for easy_install, you don't. If buildout doesn't support it, that's a > bug or limitation of buildout. Okay, then given that I'm trying to install the egg at http://svn.plone.org/svn/plone/plone.portlets/trunk which is called plone.portlets, what should the find-link and the egg name + version specifier be? Martin From pje at telecommunity.com Mon Jan 29 02:21:39 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 28 Jan 2007 20:21:39 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> At 10:52 PM 1/28/2007 +0000, Martin Aspeli wrote: >Phillip J. Eby wrote: > > At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: > >> Is there a way to do find-links in buildout? I think so, maybe named > >> something like find-links. Then you'd point at a page that has the link > >> with #egg=.... IIRC you have to point find-links at a real HTML page. > > > > Not for easy_install, you don't. If buildout doesn't support it, that's a > > bug or limitation of buildout. > >Okay, then given that I'm trying to install the egg at >http://svn.plone.org/svn/plone/plone.portlets/trunk which is called >plone.portlets, what should the find-link and the egg name + version >specifier be? If you look at the setup.py and setup.cfg that are there, it would appear you want to use #egg=plone.portlets-0.1dev as your tag and plone.portlets>=0.1dev as your requirement. From optilude at gmx.net Mon Jan 29 03:02:31 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 02:02:31 +0000 Subject: [Distutils] zc.buildout likes to download eggs... Message-ID: ... which is great, but say I have 10 buildouts all using all the eggs from zope and plone (either explicitly or as dependencies). I'd rather not replicate this in every buildout. I think there are two concerns here: - bandwidth: My ISP gets funny when I go over 20Gb/mo - disk space: A lot of information could be duplicated Are there any facilities to share eggs among packages (I have a feeling Jim implied there isn't yet)? If not, what would such a facility look like? For the bandwidth part, perhaps it could be some kind of local repository of eggs (but how to stop zc.buildout being over-zealous and getting eggs again thinking something may be newer?). For the disk space part, I'm guessing we'd need a shared "eggs" directory. I imagine I could specify eggs-directory as an absolute or relative path, e.g. ../shared-eggs. I don't know whether one buildout would clear out eggs from another one and so on in this scenario, though. Martin From optilude at gmx.net Mon Jan 29 03:21:52 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 02:21:52 +0000 Subject: [Distutils] zc.buildout likes to download eggs... In-Reply-To: <57028C04-0AD1-4941-BC1D-7B80645A9954@integrateddevcorp.com> References: <57028C04-0AD1-4941-BC1D-7B80645A9954@integrateddevcorp.com> Message-ID: <45BD5A40.2010603@gmx.net> Lists In at IDC wrote: >> Are there any facilities to share eggs among packages (I have a >> feeling >> Jim implied there isn't yet)? >> >> If not, what would such a facility look like? > > Yikes! > > I'm looking at this from a CPAN perspective and wondering whether > it's even possible that these types of issues are not yet handled by > all the work that's gone into eggs et al. You're misunderstanding. This has nothing to do with regular python packages using eggs (which they share just fine), it is about a specific tool (zc.buildout) which manages a very tightly controlled environment (for repeatable deployments). It may also be that zc.buildout does support this (which is basically useful mostly for development where you have lots of projects going on at once all using various packages), I'm only asking. Martin From jim at zope.com Mon Jan 29 12:26:52 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:26:52 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> Message-ID: <45BDD9FC.3060303@zope.com> Martin Aspeli wrote: > Phillip J. Eby wrote: >> At 12:21 PM 11/19/2006 -0500, Nathan R. Yergler wrote: >>> I'm using zc.buildout for deploying a handful of applications, and one >>> of our dependencies doesn't have eggs or "correctly" formatted pages >>> (that would allow easy_install to track down the repository for direct >>> building from source). Does zc.buildout provide any way to build an egg >> >from a Subversion URL? Something with similar behavior to: >>> easy_install -m -d ./eggs http://path/to/svn/repos >>> >>> I'm happy to write a recipe if necessary, just thought I'd make sure >>> someone else hadn't already scratched this itch. >> If it's a dependency declared in your setup.py, you can do something like this: >> >> dependency_links=['http://path/to/svn/repos#egg=Foo-dev'], >> install_requires=['Foo>=1.2,==dev'], >> >> in your setup() call, to depend on version 1.2 or greater of Foo; the ==dev >> and the #egg=Foo-dev will allow it to be found via SVN. >> >> See also these two manual sections that document the use of ==dev and #egg=: >> >> http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion >> >> http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall I need to: - try this myself, and - work up some documentation on using this with buildout. I'll add this to my to-do list. > I found myself searching for this very thing, and couldn't find a way to > do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a > syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL > in 'find-links' didn't work either. I think you got lots of hints in various responses, although I'm not sure you have the setup foo to make semse of them. :) > I'm not sure I understand zc.recipe.egg and setuptools well enough to be > able to guess what may be wrong. As Ian pointed out, this doesn't seem to be a buildout thing, although I can't be sure until I try to make it work myself. > Since zc.buildout uses setuptools > though, if there is a standard setuptools way of doing this, I'd hope it > should be possible to do it in zc.buildout? I think it should be, although I'm not sure it will have the semantics you want, although I won't be sure until I try it myself. I will try to do this this week. Feel free to nag me. :) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 29 12:27:45 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:27:45 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <45BD22C2.7030403@colorstudy.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> Message-ID: <45BDDA31.70702@zope.com> Ian Bicking wrote: > Martin Aspeli wrote: >>> http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion >>> >>> http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall >> I found myself searching for this very thing, and couldn't find a way to >> do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a >> syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL >> in 'find-links' didn't work either. >> >> I'm not sure I understand zc.recipe.egg and setuptools well enough to be >> able to guess what may be wrong. Since zc.buildout uses setuptools >> though, if there is a standard setuptools way of doing this, I'd hope it >> should be possible to do it in zc.buildout? > > Is there a way to do find-links in buildout? I think so, maybe named > something like find-links. Yup. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 29 12:29:41 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:29:41 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> Message-ID: <45BDDAA5.9070108@zope.com> Martin Aspeli wrote: > Ian Bicking wrote: > >> Is there a way to do find-links in buildout? I think so, maybe named >> something like find-links. > > Yep. > >> Then you'd point at a page that has the link >> with #egg=.... IIRC you have to point find-links at a real HTML page. > > Right. But I don't have such a page. Basically, I want my buildout to > track e.g.: > > http://svn.plone.org/svn/plone/plone.portlets/trunk > > This having a setup.py and I hope other necessary info to check it out > and build it. > > What would I need in terms of find-links for this to work? I think you need to create such a page. >> I sometimes put on in svn (maybe actually in the buildout), and point it >> there. Annoying part is you have to commit it to test. buildout, kind >> of like install_requires, I think needs you to give it something it can >> actually fulfill. So you'd need Foo==dev,>1.1 or somesuch. > > You mean in buildout's list of eggs-to-install? Basically, my > buildout.cfg now has: > > find-links = > http://download.zope.org/distribution/ > http://effbot.org/downloads > > eggs = > elementtree > python-yadis > python-openid > python-urljr > > I believe I can do foo==dev or whatever in that list of eggs as well. > But I'm not sure how that relates to svn link above. There seems to need to be a page you can point to with an #dev anchor that specified the project name. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 29 12:30:27 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:30:27 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> References: <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> Message-ID: <45BDDAD3.1050009@zope.com> Phillip J. Eby wrote: > At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: >> Is there a way to do find-links in buildout? I think so, maybe named >> something like find-links. Then you'd point at a page that has the link >> with #egg=.... IIRC you have to point find-links at a real HTML page. > > Not for easy_install, you don't. If buildout doesn't support it, that's a > bug or limitation of buildout. buildout does support find-links (and index, for that matter). Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 29 12:35:52 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:35:52 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> References: <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <5.1.1.6.0.20070128172603.05a79a78@sparrow.telecommunity.com> Message-ID: <45BDDC18.9060909@zope.com> Phillip J. Eby wrote: > At 10:01 PM 1/28/2007 +0000, Martin Aspeli wrote: >> I found myself searching for this very thing, and couldn't find a way to >> do it with zc.buildout. Listing an svn URL in the 'eggs' section gave a >> syntax error (with or without the #egg=Foo-dev bit). Listing the svn URL >> in 'find-links' didn't work either. > > Hm. easy_install accepts svn: URLs in --find-links as long as they have > the #egg part, so that sounds like a problem with zc.buildout. buildout uses setuptools.package_index.PackageIndex to find and download distributions. Does that support svn urls? Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From jim at zope.com Mon Jan 29 12:54:44 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 06:54:44 -0500 Subject: [Distutils] zc.buildout likes to download eggs... In-Reply-To: References: Message-ID: <45BDE084.7030404@zope.com> Martin Aspeli wrote: > ... which is great, but say I have 10 buildouts all using all the eggs > from zope and plone (either explicitly or as dependencies). I'd rather > not replicate this in every buildout. > > I think there are two concerns here: > > - bandwidth: My ISP gets funny when I go over 20Gb/mo > - disk space: A lot of information could be duplicated > > Are there any facilities to share eggs among packages (I have a feeling > Jim implied there isn't yet)? Yes > If not, what would such a facility look like? See below. > For the bandwidth part, perhaps it could be some kind of local > repository of eggs (but how to stop zc.buildout being over-zealous and > getting eggs again thinking something may be newer?). By default, buildout tries to get the newest distributions that meet your requirements. This is not a bandwidth issue, but given how slow PyPI is, it is a time issue. (Note that if all of your requirements have upper bounds and you have distributions that match the upper bounds, then no network access is performed). Buildout has an offline mode that prevents checks for newer versions, but also prevents getting new distributions to meet new requirements. I've decided to add a "fast" mode that skips the check for newer versions if existing local distributions satisfy requirements. > For the disk space part, I'm guessing we'd need a shared "eggs" > directory. I imagine I could specify eggs-directory as an absolute or > relative path, e.g. ../shared-eggs. I don't know whether one buildout > would clear out eggs from another one and so on in this scenario, though. You can arrange for multiple buildouts to share a common eggs directory. This is a major win both for space and speed since we can usually satisfy requirements from a shared eggs directory. The easiest way to do this is through the user-default configuration file. Buildout normally reads default configuration options from .buildout/default.cfg in your home directory (aka ~/.buildout/default.cfg). Typically people specify the buildout eggs-directory option in this file. For example, my ~/.buildout/default.cfg file looks like this: [buildout] eggs-directory = /home/jim/.buildout/eggs I had to manually create /home/jim/.buildout/eggs when I set this up. All of my buildouts look for eggs in and download eggs to this shared directory. (Note that this directory isn't used for develop eggs or eggs that are build with custom build options. It is only used for eggs that are known to be constant for a given version and thus sharable accross projects.) Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From optilude at gmx.net Mon Jan 29 22:31:38 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 21:31:38 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 10:52 PM 1/28/2007 +0000, Martin Aspeli wrote: >> Phillip J. Eby wrote: >>> At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: >>>> Is there a way to do find-links in buildout? I think so, maybe named >>>> something like find-links. Then you'd point at a page that has the link >>>> with #egg=.... IIRC you have to point find-links at a real HTML page. >>> Not for easy_install, you don't. If buildout doesn't support it, that's a >>> bug or limitation of buildout. >> Okay, then given that I'm trying to install the egg at >> http://svn.plone.org/svn/plone/plone.portlets/trunk which is called >> plone.portlets, what should the find-link and the egg name + version >> specifier be? > > If you look at the setup.py and setup.cfg that are there, it would appear > you want to use #egg=plone.portlets-0.1dev as your tag and > plone.portlets>=0.1dev as your requirement. You're absolutely right. I finally got this working now. :) The syntax in buildout.cfg is: [buildout] parts = instance zopepy # XXX: The links below are needed to track unreleased eggs; once they are # in the cheeseshop we can remove all but the first two find-links = http://download.zope.org/distribution/ http://effbot.org/downloads http://codespeak.net/svn/kukit/kss.core/trunk#egg=kss.core-0.1dev ... # Add additional eggs here eggs = elementtree python-yadis python-openid python-urljr ... kss.core ... Thank you! Martin From jim at zope.com Mon Jan 29 22:46:51 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 16:46:51 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> Message-ID: <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> On Jan 29, 2007, at 4:31 PM, Martin Aspeli wrote: > Phillip J. Eby wrote: >> At 10:52 PM 1/28/2007 +0000, Martin Aspeli wrote: >>> Phillip J. Eby wrote: >>>> At 04:25 PM 1/28/2007 -0600, Ian Bicking wrote: >>>>> Is there a way to do find-links in buildout? I think so, maybe >>>>> named >>>>> something like find-links. Then you'd point at a page that has >>>>> the link >>>>> with #egg=.... IIRC you have to point find-links at a real >>>>> HTML page. >>>> Not for easy_install, you don't. If buildout doesn't support >>>> it, that's a >>>> bug or limitation of buildout. >>> Okay, then given that I'm trying to install the egg at >>> http://svn.plone.org/svn/plone/plone.portlets/trunk which is called >>> plone.portlets, what should the find-link and the egg name + version >>> specifier be? >> >> If you look at the setup.py and setup.cfg that are there, it would >> appear >> you want to use #egg=plone.portlets-0.1dev as your tag and >> plone.portlets>=0.1dev as your requirement. > > You're absolutely right. I finally got this working now. :) Yay! I'm glad you figured this out. > The syntax in buildout.cfg is: > > [buildout] > > parts = > instance > zopepy > > # XXX: The links below are needed to track unreleased eggs; once > they are > # in the cheeseshop we can remove all but the first two > find-links = > http://download.zope.org/distribution/ > http://effbot.org/downloads > > http://codespeak.net/svn/kukit/kss.core/trunk#egg=kss.core-0.1dev > ... > > # Add additional eggs here > eggs = > elementtree > python-yadis > python-openid > python-urljr > ... > kss.core > ... > > Thank you! So do you get a develop egg and egg link or a regular egg for each of these packages? If the later, as I fear, I think you have a problem. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From optilude at gmx.net Mon Jan 29 22:52:47 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 21:52:47 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> Message-ID: <45BE6CAF.3010206@gmx.net> Jim Fulton wrote: > So do you get a develop egg and egg link or a regular egg for each of > these packages? If the later, as I fear, I think you have a problem. I think the latter (I *just* deleted my test instance and I don't want to do it all again now), but why is it a problem? Martin From jim at zope.com Mon Jan 29 23:07:36 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 17:07:36 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <45BE6CAF.3010206@gmx.net> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> Message-ID: <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> On Jan 29, 2007, at 4:52 PM, Martin Aspeli wrote: > Jim Fulton wrote: > >> So do you get a develop egg and egg link or a regular egg for each of >> these packages? If the later, as I fear, I think you have a problem. > > I think the latter (I *just* deleted my test instance and I don't want > to do it all again now), but why is it a problem? So, from your example, you now have a kss.core-0.1dev egg. It isn't a real release. It reflects the state of your subversion repository at the time you happened to run the buildout. My guess is that if you run the buildout again, the egg won't be recreated. You won't be tracking subversion changes as you wished. Further, if you use a shared eggs directory, any other buildouts will likely use that same egg, which has a more or less random snapshot of the software. I suggest that this is a bad thing. IMO, this is what you really want: - You want something to check out your project and make it a develop egg. - When you run buildout later, you want something to do an svn up in the checkout and to rerun setup (in case there are extensions). I think this "something" should probably be a recipe (or a feature of some existing recipe). I wonder what easy_install does in a case like this. For example, I wonder if easy_install would download and rebuild the egg every time you ran it. Of course, with easy_install, you probably would only run it once, but, again, then you wouldn't be tracking subversion. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From optilude at gmx.net Mon Jan 29 23:14:25 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 22:14:25 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> Message-ID: <45BE71C1.1070703@gmx.net> Jim Fulton wrote: > So, from your example, you now have a kss.core-0.1dev egg. It isn't > a real release. It reflects the state of your subversion repository > at the time you happened to run the buildout. > > My guess is that if you run the buildout again, the egg won't be > recreated. You won't be tracking subversion changes as you wished. > > Further, if you use a shared eggs directory, any other buildouts will > likely use that same egg, which has a more or less random snapshot of > the software. I suggest that this is a bad thing. > > IMO, this is what you really want: > > - You want something to check out your project and make it a develop > egg. > > - When you run buildout later, you want something to do an svn up in > the checkout and to rerun setup (in case there are extensions). > > I think this "something" should probably be a recipe (or a feature of > some existing recipe). > > I wonder what easy_install does in a case like this. For example, I > wonder if easy_install would download and rebuild the egg every time > you ran it. Of course, with easy_install, you probably would only > run it once, but, again, then you wouldn't be tracking subversion. I'd need to test this, but I *think* that it considers the svn revision as part of the "version number" so that when the svn revision changes, if buildout goes online to check for a new revision, it'd re-download and re-build. That sounds like what I want, anyway. :) Martin From optilude at gmx.net Mon Jan 29 23:14:25 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 22:14:25 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> Message-ID: <45BE71C1.1070703@gmx.net> Jim Fulton wrote: > So, from your example, you now have a kss.core-0.1dev egg. It isn't > a real release. It reflects the state of your subversion repository > at the time you happened to run the buildout. > > My guess is that if you run the buildout again, the egg won't be > recreated. You won't be tracking subversion changes as you wished. > > Further, if you use a shared eggs directory, any other buildouts will > likely use that same egg, which has a more or less random snapshot of > the software. I suggest that this is a bad thing. > > IMO, this is what you really want: > > - You want something to check out your project and make it a develop > egg. > > - When you run buildout later, you want something to do an svn up in > the checkout and to rerun setup (in case there are extensions). > > I think this "something" should probably be a recipe (or a feature of > some existing recipe). > > I wonder what easy_install does in a case like this. For example, I > wonder if easy_install would download and rebuild the egg every time > you ran it. Of course, with easy_install, you probably would only > run it once, but, again, then you wouldn't be tracking subversion. I'd need to test this, but I *think* that it considers the svn revision as part of the "version number" so that when the svn revision changes, if buildout goes online to check for a new revision, it'd re-download and re-build. That sounds like what I want, anyway. :) Martin From jim at zope.com Mon Jan 29 23:23:26 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 17:23:26 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <45BE71C1.1070703@gmx.net> References: <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <45BE71C1.1070703@gmx.net> Message-ID: On Jan 29, 2007, at 5:14 PM, Martin Aspeli wrote: > Jim Fulton wrote: > >> So, from your example, you now have a kss.core-0.1dev egg. It isn't >> a real release. It reflects the state of your subversion repository >> at the time you happened to run the buildout. >> >> My guess is that if you run the buildout again, the egg won't be >> recreated. You won't be tracking subversion changes as you wished. >> >> Further, if you use a shared eggs directory, any other buildouts will >> likely use that same egg, which has a more or less random snapshot of >> the software. I suggest that this is a bad thing. >> >> IMO, this is what you really want: >> >> - You want something to check out your project and make it a develop >> egg. >> >> - When you run buildout later, you want something to do an svn up in >> the checkout and to rerun setup (in case there are extensions). >> >> I think this "something" should probably be a recipe (or a feature of >> some existing recipe). >> >> I wonder what easy_install does in a case like this. For example, I >> wonder if easy_install would download and rebuild the egg every time >> you ran it. Of course, with easy_install, you probably would only >> run it once, but, again, then you wouldn't be tracking subversion. > > I'd need to test this, but I *think* that it considers the svn > revision > as part of the "version number" so that when the svn revision changes, I wonder if the egg it generated has the version # in it. > > if buildout goes online to check for a new revision, it'd re-download > and re-build. That sounds like what I want, anyway. :) Well, it would be an improvement, but it still might not be what you want. After all, it will check out the entire project everytime, when it would be quicker to just do an svn up. In any case, you should experiment with both easy_install and buildout. I'm fairly sure that buildout won't pick up new versions -- but I could be wrong. easy_install won't pick up new versions unless you use -U, but I wonder if it will even then. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Mon Jan 29 23:26:05 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 29 Jan 2007 17:26:05 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> Message-ID: <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> At 05:07 PM 1/29/2007 -0500, Jim Fulton wrote: >On Jan 29, 2007, at 4:52 PM, Martin Aspeli wrote: > > > Jim Fulton wrote: > > > >> So do you get a develop egg and egg link or a regular egg for each of > >> these packages? If the later, as I fear, I think you have a problem. > > > > I think the latter (I *just* deleted my test instance and I don't want > > to do it all again now), but why is it a problem? > >So, from your example, you now have a kss.core-0.1dev egg. Nope - it'll be 0.1dev-rFOO where FOO is the repository revision in question. >I wonder what easy_install does in a case like this. For example, I >wonder if easy_install would download and rebuild the egg every time >you ran it. Only if it thought the tagged version was a better match for what you were looking for. > Of course, with easy_install, you probably would only >run it once, but, again, then you wouldn't be tracking subversion. Development snapshots of setuptools can be checked out with "setuptools==dev", and that will always download a new version because the installed version is always a version like 0.6c2dev-r51290 or something. This doesn't match ==dev, thereby forcing easy_install to download a fresh copy. From jim at zope.com Mon Jan 29 23:37:28 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 17:37:28 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> Message-ID: <0826CB86-0F88-4452-B940-05C81D588D76@zope.com> On Jan 29, 2007, at 5:26 PM, Phillip J. Eby wrote: > At 05:07 PM 1/29/2007 -0500, Jim Fulton wrote: > >> On Jan 29, 2007, at 4:52 PM, Martin Aspeli wrote: >> >> > Jim Fulton wrote: >> > >> >> So do you get a develop egg and egg link or a regular egg for >> each of >> >> these packages? If the later, as I fear, I think you have a >> problem. >> > >> > I think the latter (I *just* deleted my test instance and I >> don't want >> > to do it all again now), but why is it a problem? >> >> So, from your example, you now have a kss.core-0.1dev egg. > > Nope - it'll be 0.1dev-rFOO where FOO is the repository revision in > question. Ah, good. I hope that's what he got from buildout. :) >> I wonder what easy_install does in a case like this. For example, I >> wonder if easy_install would download and rebuild the egg every time >> you ran it. > > Only if it thought the tagged version was a better match for what > you were looking for. How would it determine that? Would it try to find the current svn revision #? >> Of course, with easy_install, you probably would only >> run it once, but, again, then you wouldn't be tracking subversion. > > Development snapshots of setuptools can be checked out with > "setuptools==dev", and that will always download a new version > because the installed version is always a version like 0.6c2dev- > r51290 or something. This doesn't match ==dev, thereby forcing > easy_install to download a fresh copy. I wonder if buildout behaves this way. Buildout will ask a PackageIndex for the best available version using the obtain method. I wonder what a package index will return in a case like this. I guess I expect he'll get a version of ==dev which, as you say, will be considered newer and cause buildout to get a new distribution. Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From optilude at gmx.net Mon Jan 29 23:37:36 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 22:37:36 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > Development snapshots of setuptools can be checked out with > "setuptools==dev", and that will always download a new version because the > installed version is always a version like 0.6c2dev-r51290 or > something. This doesn't match ==dev, thereby forcing easy_install to > download a fresh copy. Could you explain how the find-link would work with an ==dev version specification? Again, this may be a buildout issue, but I have: find-links = ... http://svn.plone.org/svn/archetypes/archetypes.kss/trunk#egg=archetypes.kss-0.1dev ... eggs = ... archetypes.kss==dev ... This gives me an error: Traceback (most recent call last): File "./bin/buildout", line 12, in ? zc.buildout.buildout.main() File "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/buildout.py", line 1056, in main File "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/buildout.py", line 309, in install File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/z2c.recipe.zope2instance-0.1dev_r12104-py2.4.egg/z2c/recipe/zope2instance/__init__.py", line 36, in install requirements, ws = self.egg.working_set() File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/zc.recipe.egg-1.0.0b4-py2.4.egg/zc/recipe/egg/egg.py", line 79, in working_set path=[options['develop-eggs-directory']] File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", line 355, in install File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", line 294, in _get_dist ValueError: ("Couldn't find", Requirement.parse('archetypes.kss==dev')) I have a feeling this means my find-link is wrong, but I couldn't find a permutation that works... Martin From optilude at gmx.net Mon Jan 29 23:43:55 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 22:43:55 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <0826CB86-0F88-4452-B940-05C81D588D76@zope.com> References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> <0826CB86-0F88-4452-B940-05C81D588D76@zope.com> Message-ID: Jim Fulton wrote: > I wonder if buildout behaves this way. Buildout will ask a > PackageIndex for the best available version using the obtain method. > I wonder what a package index will return in a case like this. I > guess I expect he'll get a version of ==dev which, as you say, will > be considered newer and cause buildout to get a new distribution. Yeah, I think that's what I want. However, if I put eggname==dev in the eggs option, I get a resolution error. See my last reply to PJE's last reply to me a few minutes ago. I've confirmed that if I just use find-links as above and then list the egg name without a version spec, it does *not* try to re-download (at least as far as I can tell). Martin From jim at zope.com Mon Jan 29 23:52:50 2007 From: jim at zope.com (Jim Fulton) Date: Mon, 29 Jan 2007 17:52:50 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> Message-ID: <39916DAC-8FB7-4F7C-8830-4CC81729F144@zope.com> On Jan 29, 2007, at 5:37 PM, Martin Aspeli wrote: > Phillip J. Eby wrote: > >> Development snapshots of setuptools can be checked out with >> "setuptools==dev", and that will always download a new version >> because the >> installed version is always a version like 0.6c2dev-r51290 or >> something. This doesn't match ==dev, thereby forcing easy_install to >> download a fresh copy. > > Could you explain how the find-link would work with an ==dev version > specification? > > Again, this may be a buildout issue, but I have: > > find-links = > ... > > http://svn.plone.org/svn/archetypes/archetypes.kss/ > trunk#egg=archetypes.kss-0.1dev > ... > > eggs = > ... > archetypes.kss==dev > ... > > This gives me an error: > > Traceback (most recent call last): > File "./bin/buildout", line 12, in ? > zc.buildout.buildout.main() > File > "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/ > buildout.py", > line 1056, in main > File > "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/ > buildout.py", > line 309, in install > File > "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ > test/bot/eggs/z2c.recipe.zope2instance-0.1dev_r12104-py2.4.egg/z2c/ > recipe/zope2instance/__init__.py", > line 36, in install > requirements, ws = self.egg.working_set() > File > "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ > test/bot/eggs/zc.recipe.egg-1.0.0b4-py2.4.egg/zc/recipe/egg/egg.py", > line 79, in working_set > path=[options['develop-eggs-directory']] > File > "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ > test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", > line 355, in install > > File > "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ > test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", > line 294, in _get_dist > > ValueError: ("Couldn't find", Requirement.parse > ('archetypes.kss==dev')) > > I have a feeling this means my find-link is wrong, but I couldn't > find a > permutation that works... I think you don't need/want the the ==dev in the requirement. That is: eggs = archetypes.kss Jim -- Jim Fulton mailto:jim at zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Jan 30 00:03:31 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 29 Jan 2007 18:03:31 -0500 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: References: <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070129180219.045646e0@sparrow.telecommunity.com> At 10:37 PM 1/29/2007 +0000, Martin Aspeli wrote: >Phillip J. Eby wrote: > > > Development snapshots of setuptools can be checked out with > > "setuptools==dev", and that will always download a new version because the > > installed version is always a version like 0.6c2dev-r51290 or > > something. This doesn't match ==dev, thereby forcing easy_install to > > download a fresh copy. > >Could you explain how the find-link would work with an ==dev version >specification? #egg=archetypes.kss-dev That is, use 'dev' as the version in the link. The combination of an ==dev version specification and a #egg=projectname-dev tag means that the code will be downloaded and rebuilt *every* time, however. From optilude at gmx.net Tue Jan 30 00:02:41 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 23:02:41 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <39916DAC-8FB7-4F7C-8830-4CC81729F144@zope.com> References: <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> <39916DAC-8FB7-4F7C-8830-4CC81729F144@zope.com> Message-ID: Jim Fulton wrote: > On Jan 29, 2007, at 5:37 PM, Martin Aspeli wrote: > >> Phillip J. Eby wrote: >> >>> Development snapshots of setuptools can be checked out with >>> "setuptools==dev", and that will always download a new version >>> because the >>> installed version is always a version like 0.6c2dev-r51290 or >>> something. This doesn't match ==dev, thereby forcing easy_install to >>> download a fresh copy. >> Could you explain how the find-link would work with an ==dev version >> specification? >> >> Again, this may be a buildout issue, but I have: >> >> find-links = >> ... >> >> http://svn.plone.org/svn/archetypes/archetypes.kss/ >> trunk#egg=archetypes.kss-0.1dev >> ... >> >> eggs = >> ... >> archetypes.kss==dev >> ... >> >> This gives me an error: >> >> Traceback (most recent call last): >> File "./bin/buildout", line 12, in ? >> zc.buildout.buildout.main() >> File >> "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/ >> buildout.py", >> line 1056, in main >> File >> "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/ >> buildout.py", >> line 309, in install >> File >> "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ >> test/bot/eggs/z2c.recipe.zope2instance-0.1dev_r12104-py2.4.egg/z2c/ >> recipe/zope2instance/__init__.py", >> line 36, in install >> requirements, ws = self.egg.working_set() >> File >> "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ >> test/bot/eggs/zc.recipe.egg-1.0.0b4-py2.4.egg/zc/recipe/egg/egg.py", >> line 79, in working_set >> path=[options['develop-eggs-directory']] >> File >> "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ >> test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", >> line 355, in install >> >> File >> "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/ >> test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", >> line 294, in _get_dist >> >> ValueError: ("Couldn't find", Requirement.parse >> ('archetypes.kss==dev')) >> >> I have a feeling this means my find-link is wrong, but I couldn't >> find a >> permutation that works... > > I think you don't need/want the the ==dev in the requirement. > That is: > > eggs = archetypes.kss Yes, that works, but then it doesn't seem to re-check svn for a new version each time. Martin From optilude at gmx.net Tue Jan 30 00:14:15 2007 From: optilude at gmx.net (Martin Aspeli) Date: Mon, 29 Jan 2007 23:14:15 +0000 Subject: [Distutils] Building an egg from Subversion using zc.buildout In-Reply-To: <5.1.1.6.0.20070129180219.045646e0@sparrow.telecommunity.com> References: <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> <45BE6CAF.3010206@gmx.net> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <1163956881.5763.21.camel@Jane> <5.1.1.6.0.20061119134913.01f3d088@sparrow.telecommunity.com> <45BD22C2.7030403@colorstudy.com> <5.1.1.6.0.20070128174203.042de500@sparrow.telecommunity.com> <5.1.1.6.0.20070128202027.04461008@sparrow.telecommunity.com> <9D0FD265-B323-42C3-80A1-85638A067C9A@zope.com> <45BE6CAF.3010206@gmx.net> <65788D22-5FA7-4AC1-A6FA-08FDA4D173FE@zope.com> <5.1.1.6.0.20070129172214.04469590@sparrow.telecommunity.com> <5.1.1.6.0.20070129180219.045646e0@sparrow.telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 10:37 PM 1/29/2007 +0000, Martin Aspeli wrote: >> Phillip J. Eby wrote: >> >>> Development snapshots of setuptools can be checked out with >>> "setuptools==dev", and that will always download a new version because the >>> installed version is always a version like 0.6c2dev-r51290 or >>> something. This doesn't match ==dev, thereby forcing easy_install to >>> download a fresh copy. >> Could you explain how the find-link would work with an ==dev version >> specification? > > #egg=archetypes.kss-dev > > That is, use 'dev' as the version in the link. The combination of an ==dev > version specification and a #egg=projectname-dev tag means that the code > will be downloaded and rebuilt *every* time, however. Alas, this gives me this error: Traceback (most recent call last): File "./bin/buildout", line 12, in ? zc.buildout.buildout.main() File "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/buildout.py", line 1056, in main File "/private/tmp/tmp29h1vY/zc.buildout-1.0.0b19-py2.4.egg/zc/buildout/buildout.py", line 309, in install File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/z2c.recipe.zope2instance-0.1dev_r12104-py2.4.egg/z2c/recipe/zope2instance/__init__.py", line 36, in install requirements, ws = self.egg.working_set() File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/zc.recipe.egg-1.0.0b4-py2.4.egg/zc/recipe/egg/egg.py", line 79, in working_set path=[options['develop-eggs-directory']] File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", line 355, in install File "/Users/optilude/Development/Plone/Code/Products/ZopeSkel/trunk/test/bot/eggs/setuptools-0.6c5-py2.4.egg/easy_install.py", line 294, in _get_dist ValueError: ("Couldn't find", Requirement.parse('archetypes.kss==dev')) Is there something else I need to do to the svn repository (http://svn.plone.org/svn/archetypes/archetypes.kss/trunk)? Or is this a zc.buildout limitation? Martin From howard at eegsoftware.com Fri Jan 19 20:26:21 2007 From: howard at eegsoftware.com (Howard Lightstone) Date: Fri, 19 Jan 2007 11:26:21 -0800 Subject: [Distutils] Compiler support for VC++ 2005 Message-ID: <45B0AADD.13806.25AF95@howard.eegsoftware.com> I am unable to rebuild things using disutils using Visual Studio 2005. I get reports of "No compiler found". Since VC8 is the only thing available in the future (from Micro$soft), is there any progress toward updating distuils to allow this compiler (and its own locations....) to be supported?