From cournape at gmail.com Mon Oct 1 00:21:35 2012 From: cournape at gmail.com (David Cournapeau) Date: Sun, 30 Sep 2012 23:21:35 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: <14DC3D7670F4434E970EDD724F1D7660@gmail.com> References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: On Sun, Sep 30, 2012 at 5:07 PM, Donald Stufft wrote: > On Sunday, September 30, 2012 at 8:59 AM, David Cournapeau wrote: > > Note that all this work has already been done in Bento. > > I understand the appeal of using an existing format like yaml, but it > is not clear to me how one can handle conditional with it, and I think > you want to handle conditionals in there (for platform-dependent > dependencies). > > Bento also has a command to convert setup.py-based projects to bento > internal format, adapting it to use another format should not be too > difficult. > > David > > Instead of conditionals the existing ideas use an environment marker, > so instead of (pseudo format, I just woke up): > > if python_version < 2.6: > require: simplejson > > You would do: > require: simplejson; python_version < 2.6 Right, in that case, it would work, but in my experience, it is important to be able to apply conditionals to more than just requirements (packages definition and so on). I am not suggesting something very complicated (we don't want to re-create a language), but more something like cabal (see conditionals in http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions), or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html). This has the advantage of using something that has already been battle-tested for several years. But again, syntax is a really minor point, and what matters the most are the features. I was not able to express what I wanted with yaml for bento, but would be more than happy to scrap the custom format if someone manages to. David From donald.stufft at gmail.com Mon Oct 1 00:35:03 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 30 Sep 2012 18:35:03 -0400 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: <82CD5AE06F1943689A07231B643BC19B@gmail.com> On Sunday, September 30, 2012 at 6:21 PM, David Cournapeau wrote: > Right, in that case, it would work, but in my experience, it is > important to be able to apply conditionals to more than just > requirements (packages definition and so on). > A significant problem is caused by the allowance of if statements in setup.py for trying to generically pull data out. Obviously the limited conditionals of Bento make that easier, but you still need to essentially maintain a matrix of every possible combination of conditions in order to accurately represent that data when you don't know what the target system is going to look at. If other fields benefit from the environment markers then by all means lets add them to other fields. A static piece of metadata that is unchanging is a powerful tool on the index side. It allows some of the current stuff i've been experimenting with to work very cleanly, such as being able to fetch an entire dependency tree in one request. Now again i'm talking solely about the in distribution format and not about however the tool the developer uses to create creates that file. So for example bento could easily keep it's conditional based processing of the info file, it would just need to be smart enough to "compact" it down to the environment markers in the standard file included with the distribution. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Oct 1 00:50:49 2012 From: cournape at gmail.com (David Cournapeau) Date: Sun, 30 Sep 2012 23:50:49 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: <82CD5AE06F1943689A07231B643BC19B@gmail.com> References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> <82CD5AE06F1943689A07231B643BC19B@gmail.com> Message-ID: On Sun, Sep 30, 2012 at 11:35 PM, Donald Stufft wrote: > On Sunday, September 30, 2012 at 6:21 PM, David Cournapeau wrote: > > Right, in that case, it would work, but in my experience, it is > important to be able to apply conditionals to more than just > requirements (packages definition and so on). > > A significant problem is caused by the allowance of if statements in > setup.py for trying to generically pull data out. Obviously the limited > conditionals of Bento make that easier, but you still need to essentially > maintain a matrix of every possible combination of conditions in order > to accurately represent that data when you don't know what the target > system is going to look at. If other fields benefit from the environment > markers then by all means lets add them to other fields. > > A static piece of metadata that is unchanging is a powerful tool on the > index side. It allows some of the current stuff i've been experimenting > with to work very cleanly, such as being able to fetch an entire dependency > tree in one request. I am not sure I understand the argument: whatever the syntax, if the feature is there, you will have the same problem ? The fact that it is used by existing solutions tend to convince me it is not a problem (cabal works much better than distutils/etc... in pretty much every way, including installing things with dependency from their package servers, not to speak about RPM). This could be abused of course, but it should not be too difficult to give suggestions in those cases (one big advantage of whatever-format-we-want compared to setup.py). Conditional markers do make the dependency harder to get right, as conditional in dependencies + provides is what makes dependency graph building complicated. It seems this can be solved neatly though (see http://www.slideshare.net/naderman/dependency-resolution-with-sat-symfony-live-2011-paris). David From donald.stufft at gmail.com Mon Oct 1 01:40:51 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Sun, 30 Sep 2012 19:40:51 -0400 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> <82CD5AE06F1943689A07231B643BC19B@gmail.com> Message-ID: <499737A0D6344B21B0688DB99AEC6DBC@gmail.com> On Sunday, September 30, 2012 at 6:50 PM, David Cournapeau wrote: > I am not sure I understand the argument: whatever the syntax, if the > feature is there, you will have the same problem ? The fact that it is > used by existing solutions tend to convince me it is not a problem > (cabal works much better than distutils/etc... in pretty much every > way, including installing things with dependency from their package > servers, not to speak about RPM). > > I have a site that pulls data from python packages, one of my problems of late has been pulling requirements data out of a setuptools setup.py. Simply fetching the data for my current system is simple, however because of the use of conditionals that affect the final outcome of the metadata in order to make sure I get a true and complete accounting of all the metadata I would have to anticipate every combination of conditions used inside that file. Now as far as I can tell, the bento.info conditionals have that same problem except that they have a much narrow scope, so they have a smaller matrix. How would I given a package that has an ``if os(windows)`` conditional, get a representation of that files metadata as if it were in windows? I would need to override what os(windows) is? Now if this same package also has conditonals for flags, the matrix becomes larger and larger and I need to either create my own custom parser that just ignores or annotates the conditionals or I need to use the official parser and run it through once per item in the matrix to get the "whole story". Am I incorrect in that? Additionally, conditionals as they are (to my understanding, having never used bento but having read the docs and some of it's source code) implemented in bento complicate the parser and confuse reading a file with understanding a file. What additional power or benefit does: Library: InstallRequires: docutils, sphinx if os(windows): pywin32 provide over: { "Library": { "InstallRequires": [ "docutils", "sphinx", "pywin32; sys.playform == win32" ] } } (Note I dislike the ; syntax too, but I dislike the conditionals more.) Maybe i'm missing some powerful construct in the bento.info that warrants throwing out a lot of hard work and testing of a lot of individuals for these common formats, however I just don't see it in merely providing a simple conditional statement. The ``bento.parser`` package contains ~1700 SLOC now without investigating that terribly deeply I'm willing to bet almost all of that could be thrown out in the use of a standard format. I know you've mentioned Cabal a lot when talking about bento, and it's obvious it's been a very big influence on your design. > > Conditional markers do make the dependency harder to get right, as > conditional in dependencies + provides is what makes dependency graph > building complicated. It seems this can be solved neatly though (see > http://www.slideshare.net/naderman/dependency-resolution-with-sat-symfony-live-2011-paris). > Good link :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Oct 1 04:00:57 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 30 Sep 2012 22:00:57 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine Message-ID: I thought it would be interesting to get an idea of the relative popularity of the various pkg_resources calls in the interest of emulating a subset. So I looked it up on Nullege. The absolute popularity is certainly different from the numbers below. I think of require() and load_entry_point() used in console scripts as implementing a dynamic linker, so for example if some code crashes when the KDEWallet bindings are installed, you would still be able to run it by omitting those from sys.path. Gem certainly works this way. I'm not 100% sure pkg_resources works that way. Those functions traverse the dependency graph at runtime which is fast because it doesn't include the "search pypi" stuff. resource_filename is very popular. I would have thought resource_stream would be more popular. Unless your package is zipped resource_filename is trivial to implement. If you are writing an installer, you can unpack a single distribution to a folder, run find_distributions() on that folder, and get a Distribution() object with the dependencies as a dict. -- Daniel H pkg_resources APIs by number of Nullege Call() counts: 'require', 2284 'run_script', 0 'get_provider', 32 'get_distribution', 103 'load_entry_point', 31 'get_entry_map', 11 'get_entry_info', 4 'iter_entry_points', 291 'resource_string', 175 'resource_stream', 155 'resource_filename', 713 'resource_listdir', 71 'resource_exists', 67 'resource_isdir', 18 'declare_namespace', 643; obsoleted by Python 3.3 or pkgutil 'working_set', 55 (all samples); not a function 'add_activation_listener', 3 'find_distributions', 25; needed by installers 'set_extraction_path', 2 'cleanup_resources', 4 'get_default_cache', 2 'Environment', 51 'WorkingSet', 17 'ResourceManager', 27 'Distribution', 22; needed by installers 'Requirement.parse', 524 'EntryPoint.parse', 45 'ResolutionError', 6 samples 'VersionConflict', 13 samples 'DistributionNotFound', 41 samples 'UnknownExtra', 8 samples 'ExtractionError', 0 'parse_requirements', 32 'parse_version', 100 'safe_name', trivial regexp replace 'safe_version', "" 'get_platform', 1 'compatible_platforms', 0 'yield_lines', 15; trivial 'split_sections', 9; .ini-style [section] parser 'safe_extra', 0; another regexp replacement 'to_filename', 9; another text replacement 'ensure_directory', 40 'normalize_path', 42 # Constants that control which kinds of dists are preferred 'EGG_DIST', 'BINARY_DIST', 'SOURCE_DIST', 'CHECKOUT_DIST', 'DEVELOP_DIST', 'IMetadataProvider', 'IResourceProvider', 'FileMetadata', 'PathMetadata', 23 'EggMetadata', 2 'EmptyProvider', 'empty_provider', 'NullProvider', subclassed a few times for unit test mocks 'EggProvider', 0 'DefaultProvider', 21 mentions 'ZipProvider', 1 subclass 'register_finder', 'register_namespace_handler', 'register_loader_type', 6; one user is a code signing library 'fixup_namespace_packages', 0 'get_importer', 0 'AvailableDistributions'] From dholth at gmail.com Mon Oct 1 05:06:52 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 30 Sep 2012 23:06:52 -0400 Subject: [Distutils] ANN wheel-0.10.1 Message-ID: I'm pleased to announce wheel version 0.10.1. In this version of wheel, Paul Moore has done a lot of work on wheel's reference installer, a simple installer that can only install from local wheel files. The wheel installer can now accept pip-style requirements files but it doesn't automatically install dependencies, and it does not uninstall conflicting packages i.e. previous versions of the same package. If you need a mature full-featured installer you should use pip. A notable feature of this version of wheel is that you can use it even when your virtualenv does not contain setuptools or distribute. For example, /tmp/venv/bin/python wheel-0.10.1-py2.py3-none-any.whl/wheel install wheel-0.10.1-py2.py3-none-any.whl executes wheel's wheel to install itself. This feature should be handy for bootstrapping. Wheel's own scripts still use pkg_resources, so you should invoke the tool with "python -m wheel" if you do not want to install distribute or setuptools. On my machine, it takes about 0.7 seconds to install IPython from a wheel. From dholth at gmail.com Mon Oct 1 05:31:12 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 30 Sep 2012 23:31:12 -0400 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: <499737A0D6344B21B0688DB99AEC6DBC@gmail.com> References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> <82CD5AE06F1943689A07231B643BC19B@gmail.com> <499737A0D6344B21B0688DB99AEC6DBC@gmail.com> Message-ID: On Sun, Sep 30, 2012 at 7:40 PM, Donald Stufft wrote: > On Sunday, September 30, 2012 at 6:50 PM, David Cournapeau wrote: > > I am not sure I understand the argument: whatever the syntax, if the > feature is there, you will have the same problem ? The fact that it is > used by existing solutions tend to convince me it is not a problem > (cabal works much better than distutils/etc... in pretty much every > way, including installing things with dependency from their package > servers, not to speak about RPM). > > I have a site that pulls data from python packages, one of my problems > of late has been pulling requirements data out of a setuptools setup.py. > Simply > fetching the data for my current system is simple, however because of the > use > of conditionals that affect the final outcome of the metadata in order to > make sure I get a true and complete accounting of all the metadata I would > have to anticipate every combination of conditions used inside that file. This is real setup.py problem and the packaging peps did a pretty good job of making this better (but no one uses environment markers yet). You might arrange to have it run on some number of common Pythons. You could also parse the setup.py with the ast module to figure out whether or not the install_requires was modified inside condition statements. You don't have to look at setup.py at all, you can just look for *.egg-info/requires.txt and you will get the requirements as they appeared on the publisher's machine. (pip always runs setup.py egg_info again, to get trustworthy metadata, while it is resolving deps). > Now as far as I can tell, the bento.info conditionals have that same problem > except that they have a much narrow scope, so they have a smaller matrix. > > How would I given a package that has an ``if os(windows)`` conditional, get > a representation of that files metadata as if it were in windows? I would > need > to override what os(windows) is? Now if this same package also has > conditonals > for flags, the matrix becomes larger and larger and I need to either create > my own custom parser that just ignores or annotates the conditionals or I > need > to use the official parser and run it through once per item in the matrix to > get > the "whole story". Am I incorrect in that? > > Additionally, conditionals as they are (to my understanding, having never > used bento but having read the docs and some of it's source code) > implemented > in bento complicate the parser and confuse reading a file with understanding > a file. > > What additional power or benefit does: > > Library: > InstallRequires: > docutils, > sphinx > if os(windows): > pywin32 > > provide over: > > { > "Library": { > "InstallRequires": [ > "docutils", > "sphinx", > "pywin32; sys.playform == win32" > ] > } > } > > (Note I dislike the ; syntax too, but I dislike the conditionals > more.) > > Maybe i'm missing some powerful construct in the bento.info that warrants > throwing out a lot of hard work and testing of a lot of individuals for > these > common formats, however I just don't see it in merely providing a simple > conditional statement. The ``bento.parser`` package contains ~1700 SLOC > now without investigating that terribly deeply I'm willing to bet almost all > of that could be thrown out in the use of a standard format. David C. would probably be the first to admit that he is not principally a compiler author. You are a Python programmer and you want to use {} instead of (indentation)? Don't answer that. He should probably produce an AST (not "import ast") from the bento.info with the conditionals, it sounds like it may currently interpret while parsing, I haven't looked. All the syntax arguments are a red herring. The other stuff "separating build phases cleanly using an on-disk format" and "extensible" is what makes Bento beautiful. Not having to write .ini files is a definite plus though. > Conditional markers do make the dependency harder to get right, as > conditional in dependencies + provides is what makes dependency graph > building complicated. It seems this can be solved neatly though (see > http://www.slideshare.net/naderman/dependency-resolution-with-sat-symfony-live-2011-paris). Well in pkg_resources the default environment for environment marker functions (only supported in .dist-info style distributions) is: _VARS = {'sys.platform': sys.platform, 'python_version': '%s.%s' % sys.version_info[:2], # FIXME parsing sys.platform is not reliable, but there is no other # way to get e.g. 2.7.2+, and the PEP is defined with sys.version 'python_full_version': sys.version.split(' ', 1)[0], 'os.name': os.name, 'platform.version': platform.version(), 'platform.machine': platform.machine(), 'platform.python_implementation': python_implementation(), 'extra': None # wheel extension } Your users could send you that dict, or you could just evaluate every environment marker all as true and send them all the pkg-resources files that they could possibly need. Unfortunately no one is using environment markers. If you would like to use environment markers for your requirements, you can include in setup.cfg [metadata] requires-dist = argparse; python_version < '2.7' another_dep and package your dist with "python setup.py bdist_wheel". The deps in [metadata] will override the install_requires argument to setup.cfg. When using the patched pip to install from the wheel file, if markerlib and distribute >= 0.6.28 are installed, pip will resolve dependencies based on those markers. From cournape at gmail.com Mon Oct 1 08:04:12 2012 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Oct 2012 07:04:12 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: <499737A0D6344B21B0688DB99AEC6DBC@gmail.com> References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> <82CD5AE06F1943689A07231B643BC19B@gmail.com> <499737A0D6344B21B0688DB99AEC6DBC@gmail.com> Message-ID: On Mon, Oct 1, 2012 at 12:40 AM, Donald Stufft wrote: > On Sunday, September 30, 2012 at 6:50 PM, David Cournapeau wrote: > > I am not sure I understand the argument: whatever the syntax, if the > feature is there, you will have the same problem ? The fact that it is > used by existing solutions tend to convince me it is not a problem > (cabal works much better than distutils/etc... in pretty much every > way, including installing things with dependency from their package > servers, not to speak about RPM). > > I have a site that pulls data from python packages, one of my problems > of late has been pulling requirements data out of a setuptools setup.py. > Simply > fetching the data for my current system is simple, however because of the > use > of conditionals that affect the final outcome of the metadata in order to > make sure I get a true and complete accounting of all the metadata I would > have to anticipate every combination of conditions used inside that file. > > Now as far as I can tell, the bento.info conditionals have that same problem > except that they have a much narrow scope, so they have a smaller matrix. > > How would I given a package that has an ``if os(windows)`` conditional, get > a representation of that files metadata as if it were in windows? I would > need > to override what os(windows) is? Now if this same package also has > conditonals > for flags, the matrix becomes larger and larger and I need to either create > my own custom parser that just ignores or annotates the conditionals or I > need > to use the official parser and run it through once per item in the matrix to > get > the "whole story". Am I incorrect in that? > > Additionally, conditionals as they are (to my understanding, having never > used bento but having read the docs and some of it's source code) > implemented > in bento complicate the parser and confuse reading a file with understanding > a file. > > What additional power or benefit does: > > Library: > InstallRequires: > docutils, > sphinx > if os(windows): > pywin32 > > provide over: > > { > "Library": { > "InstallRequires": [ > "docutils", > "sphinx", > "pywin32; sys.playform == win32" > ] > } > } It is obviously exactly the same, it is just different syntax (which is why I don't understand the "fetching requirements" arguments, since whatever is true for bento is true of your format and vice et versa). > > Maybe i'm missing some powerful construct in the bento.info that warrants > throwing out a lot of hard work and testing of a lot of individuals for > these > common formats, however I just don't see it in merely providing a simple > conditional statement. The ``bento.parser`` package contains ~1700 SLOC > now without investigating that terribly deeply I'm willing to bet almost all > of that could be thrown out in the use of a standard format. What I am afraid of is that one will need to add a lots of those markers, not just for requirements, which will lose the point of using a common format. I would also recommend against json, because it does not allow for comments, yaml is much better in that regard. > I know you've mentioned Cabal a lot when talking about bento, and it's > obvious it's been a very big influence on your design. Mostly the syntax, actually. David From vinay_sajip at yahoo.co.uk Mon Oct 1 09:14:08 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 07:14:08 +0000 (UTC) Subject: [Distutils] [Python-Dev] distlib updated with resources API References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: David Cournapeau gmail.com> writes: > I am not suggesting something very complicated (we don't want to > re-create a language), but more something like cabal (see conditionals > in http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions), > or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html). > This has the advantage of using something that has already been > battle-tested for several years. Both the Cabal and RPM conditionals seem fairly narrow in scope (e.g. os/ arch/impl/flag for Cabal), and therefore it seems that environment markers should be able to do the job. Although of course environment markers aren't battle tested, it seems worthwhile deploying them on the battlefield to see how they perform. > But again, syntax is a really minor point, and what matters the most > are the features. I was not able to express what I wanted with yaml > for bento, but would be more than happy to scrap the custom format if > someone manages to. If I can continue working on the YAML format that I've mentioned, I hope I can report some progress in this area in due course. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Mon Oct 1 09:35:20 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 07:35:20 +0000 (UTC) Subject: [Distutils] =?utf-8?q?pkg=5Fresources_API_use_according_to_Nulleg?= =?utf-8?q?e_search_engine?= References: Message-ID: Daniel Holth gmail.com> writes: > resource_filename is very popular. I would have thought > resource_stream would be more popular. Unless your package is zipped > resource_filename is trivial to implement. Yes, I find that odd, too. pkg_resources seems to extract files from zip into a cache folder, then returns filenames from the location in the cache; it seems a lot of trouble to go to just to be able to deliver a filename. In the Resource that I implemented in distlib, I have the "path" attribute of a resource which is analogous, though it's not directly usable as a file path for resources in a zip. However, since the resource is available as bytes or a stream, those applications which really need a filename (perhaps to pass to a third-party API which expects a filename) can handle that themselves, e.g. by saving the bytes to a temporary location and passing that to whatever needs a filename. Regards, Vinay Sajip From cournape at gmail.com Mon Oct 1 15:48:35 2012 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Oct 2012 14:48:35 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: On Mon, Oct 1, 2012 at 8:14 AM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> I am not suggesting something very complicated (we don't want to >> re-create a language), but more something like cabal (see conditionals >> in > http://www.haskell.org/cabal/users-guide/developing-packages.html#package-descriptions), >> or even RPM (http://www.rpm.org/max-rpm/s1-rpm-inside-conditionals.html). >> This has the advantage of using something that has already been >> battle-tested for several years. > > Both the Cabal and RPM conditionals seem fairly narrow in scope (e.g. os/ > arch/impl/flag for Cabal), and therefore it seems that environment markers > should be able to do the job. Although of course environment markers aren't > battle tested, it seems worthwhile deploying them on the battlefield to see how > they perform. Note that in Cabal at least, those conditionals work not just for requirements, but for pretty much any section that is not metadata (so in the python world, you could condition on the package you want to install, etc...). > >> But again, syntax is a really minor point, and what matters the most >> are the features. I was not able to express what I wanted with yaml >> for bento, but would be more than happy to scrap the custom format if >> someone manages to. > > If I can continue working on the YAML format that I've mentioned, I hope I can > report some progress in this area in due course. That's great. May I ask you to put this code somewhere ? This would allow me to try using this inside bento. It would allow using whatever format you are ending up with directly with existing projects (bento is is already used by some complex packages such as numpy or scipy). It would be a good way to ensure the format provides enough semantics. David From vinay_sajip at yahoo.co.uk Mon Oct 1 16:15:07 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 14:15:07 +0000 (UTC) Subject: [Distutils] [Python-Dev] distlib updated with resources API References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: David Cournapeau gmail.com> writes: > Note that in Cabal at least, those conditionals work not just for > requirements, but for pretty much any section that is not metadata (so > in the python world, you could condition on the package you want to > install, etc...). Right, but the concept of environment markers is simple enough that we should be able to extend to other areas. Requirements are just the most obvious application. > That's great. May I ask you to put this code somewhere ? This would > allow me to try using this inside bento. It would allow using whatever > format you are ending up with directly with existing projects (bento > is is already used by some complex packages such as numpy or scipy). > It would be a good way to ensure the format provides enough semantics. Which code do you mean? Although I have written some code to produce metadata in YAML format, I have not yet got anything that consumes it to a reasonable point (apart from sdist generation, which should not be environment-specific). I will publish what I have as soon as it has reached a reasonable state of usefulness. Precisely because it isn't declarative, existing environment-specific code in setup.py files in PyPI archives is not easy to convert to environment markers automatically :-( Regards, Vinay Sajip From cournape at gmail.com Mon Oct 1 16:29:29 2012 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Oct 2012 15:29:29 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: On Mon, Oct 1, 2012 at 3:15 PM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> Note that in Cabal at least, those conditionals work not just for >> requirements, but for pretty much any section that is not metadata (so >> in the python world, you could condition on the package you want to >> install, etc...). > > Right, but the concept of environment markers is simple enough that we should > be able to extend to other areas. Requirements are just the most obvious > application. I think we just need to see how it looks like in real cases to get a good idea about tradeoffs syntax-wise. > >> That's great. May I ask you to put this code somewhere ? This would >> allow me to try using this inside bento. It would allow using whatever >> format you are ending up with directly with existing projects (bento >> is is already used by some complex packages such as numpy or scipy). >> It would be a good way to ensure the format provides enough semantics. > > Which code do you mean? Although I have written some code to produce metadata in > YAML format, I have not yet got anything that consumes it to a reasonable point > (apart from sdist generation, which should not be environment-specific). The code that produces yaml files. The point is precisely that it would be easy for me to consume this and produce the internal package representation in bento, which would then allow to configure, build and install packages using the bento format. > I will publish what I have as soon as it has reached a reasonable state of > usefulness. Precisely because it isn't declarative, existing environment-specific > code in setup.py files in PyPI archives is not easy to convert to environment > markers automatically :-( I am curious: do you attempt to parse the setup.py to get those environment markers ? I personally gave up on this and just run the setup.py to get whatever values are available in that precise environment. Given that versions are not normalized, I am afraid trying to do better is bound to fail, but maybe I am not smart enough to do it :) David From barry at python.org Mon Oct 1 16:46:30 2012 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Oct 2012 10:46:30 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: References: Message-ID: <20121001104630.3e45eaf4@limelight.wooz.org> On Sep 30, 2012, at 10:00 PM, Daniel Holth wrote: >resource_filename is very popular. I would have thought >resource_stream would be more popular. Unless your package is zipped >resource_filename is trivial to implement. I've used declare_namespace() once or twice. I use resource_filename(), resource_string(), resource_stream(), and resource_listdir() quite a bit. Cheers, -Barry From barry at python.org Mon Oct 1 16:48:41 2012 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Oct 2012 10:48:41 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: References: Message-ID: <20121001104841.04693de5@limelight.wooz.org> On Oct 01, 2012, at 07:35 AM, Vinay Sajip wrote: >Yes, I find that odd, too. pkg_resources seems to extract files from zip into >a cache folder, then returns filenames from the location in the cache; it >seems a lot of trouble to go to just to be able to deliver a filename. Darn handy though when interacting with some APIs which require filenames. >In the Resource that I implemented in distlib, I have the "path" attribute of >a resource which is analogous, though it's not directly usable as a file path >for resources in a zip. However, since the resource is available as bytes or >a stream, those applications which really need a filename (perhaps to pass to >a third-party API which expects a filename) can handle that themselves, >e.g. by saving the bytes to a temporary location and passing that to whatever >needs a filename. Why not provide this by distlib? -Barry From dholth at gmail.com Mon Oct 1 16:52:24 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 1 Oct 2012 10:52:24 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: <20121001104630.3e45eaf4@limelight.wooz.org> References: <20121001104630.3e45eaf4@limelight.wooz.org> Message-ID: On Mon, Oct 1, 2012 at 10:46 AM, Barry Warsaw wrote: > On Sep 30, 2012, at 10:00 PM, Daniel Holth wrote: > >>resource_filename is very popular. I would have thought >>resource_stream would be more popular. Unless your package is zipped >>resource_filename is trivial to implement. > > I've used declare_namespace() once or twice. > > I use resource_filename(), resource_string(), resource_stream(), and > resource_listdir() quite a bit. declare_namespace() is special because we could probably rewrite it at install time (in Python 3.3, by deleting __init__.py) or by calling pkgutil. From erik.m.bray at gmail.com Mon Oct 1 17:31:42 2012 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 1 Oct 2012 11:31:42 -0400 Subject: [Distutils] Differences in PEP386 and setuptools In-Reply-To: <20120927231516.GP2974@unaka.lan> References: <20120926200028.GH2974@unaka.lan> <215BFDFF990F49E5A920022A9C8E1F47@gmail.com> <93ECE7DE9BE749399E79589762F936CE@gmail.com> <20120927155900.GI2974@unaka.lan> <20120927180335.GL2974@unaka.lan> <0371FC78A53B43FEA000E06F3F5C2BF1@gmail.com> <20120927194529.GO2974@unaka.lan> <20120927231516.GP2974@unaka.lan> Message-ID: On Thu, Sep 27, 2012 at 7:15 PM, Toshio Kuratomi wrote: > So to be more complete, the proposal is to do this:: > > 0.1 > < 0.1.post1 > < 1.0.dev1 # The .dev that's changing position > < 1.0a1.dev1 > < 1.0a1 > < 1.0a1.post1 > < 1.0b1.dev1 > < 1.0b1 > < 1.0b1.post1 > < 1.0c1.dev1 > < 1.0c1 > < 1.0c1.post1 > # The current PEP386 position of 1.0.dev1 > < 1.0 > < 1.0.post1 > This looks perfect to me, and very unambiguous. Allowing the .dev and .post modifiers at every release stage I think clears up the issue here better than any other proposal. +1. (This is also semantically the same as semver--just not syntactically the same. I like semver but I think we're stuck with the existing syntax which is fine.) Erik From vinay_sajip at yahoo.co.uk Mon Oct 1 17:46:56 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 16:46:56 +0100 (BST) Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: <1349106416.84727.YahooMailNeo@web171403.mail.ir2.yahoo.com> Yes, and I've added its package.yaml to the original Gist at https://gist.github.com/3803556 Regards, Vinay Sajip >________________________________ > From: Daniel Holth >To: Vinay Sajip >Sent: Monday, 1 October 2012, 15:27 >Subject: Re: [Distutils] [Python-Dev] distlib updated with resources API > >Have you read the IPython setup.py? >https://github.com/ipython/ipython/blob/master/setup.py > > > From vinay_sajip at yahoo.co.uk Mon Oct 1 17:59:23 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 16:59:23 +0100 (BST) Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: <7E31FFBE-A805-497E-ABC9-52E5845600FE@gmail.com> References: <7E31FFBE-A805-497E-ABC9-52E5845600FE@gmail.com> Message-ID: <1349107163.376.YahooMailNeo@web171406.mail.ir2.yahoo.com> > > Of course it's not really a mystery; a tremendous number of Apis only accept > a file name. That's why it's there. And only something well connected to > the importer system knows how and whether the extraction is needed. > Why is that the case? If the bytes are available, then whoever needs can write them to a temp file, pass that name, and clean up afterwards. Is managing a cache of arbitrary files extracted from zips is a mainstream use case which distlib needs to support? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Mon Oct 1 17:43:19 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 15:43:19 +0000 (UTC) Subject: [Distutils] =?utf-8?q?pkg=5Fresources_API_use_according_to_Nulleg?= =?utf-8?q?e_search_engine?= References: <20121001104841.04693de5@limelight.wooz.org> Message-ID: Barry Warsaw python.org> writes: > >a third-party API which expects a filename) can handle that themselves, > >e.g. by saving the bytes to a temporary location and passing that to whatever > >needs a filename. > > Why not provide this by distlib? Only because it doesn't add much value, and (if you do all that pkg_resources does) might mean that you have to maintain a cache. As a matter of interest, what are the APIs you're using which need filenames but can't use bytes or streams? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Mon Oct 1 18:06:11 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 16:06:11 +0000 (UTC) Subject: [Distutils] [Python-Dev] distlib updated with resources API References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: David Cournapeau gmail.com> writes: > The code that produces yaml files. The point is precisely that it > would be easy for me to consume this and produce the internal package > representation in bento, which would then allow to configure, build > and install packages using the bento format. Well, it's not pretty, but here's a Gist of the method which produces YAML metadata from what's passed to setup(): https://gist.github.com/3812561 > I am curious: do you attempt to parse the setup.py to get those > environment markers ? I personally gave up on this and just run the > setup.py to get whatever values are available in that precise Lord, no, life's too short :-) Regards, Vinay Sajip From barry at python.org Mon Oct 1 18:18:20 2012 From: barry at python.org (Barry Warsaw) Date: Mon, 1 Oct 2012 12:18:20 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: References: <20121001104841.04693de5@limelight.wooz.org> Message-ID: <20121001121820.478adabf@limelight.wooz.org> On Oct 01, 2012, at 03:43 PM, Vinay Sajip wrote: >Barry Warsaw python.org> writes: >> Why not provide this by distlib? > >Only because it doesn't add much value, and (if you do all that pkg_resources >does) might mean that you have to maintain a cache. Well, it's just something that client code would have to do anyway, which leads to multiple wheel invention and bugs. >As a matter of interest, what are the APIs you're using which need filenames >but can't use bytes or streams? Two that come to mind. * Testing command line arguments that take a file name (e.g. test that --config works). * Generating file: urls to generalized downloaders for local resources. Cheers, -Barry From cournape at gmail.com Mon Oct 1 20:33:05 2012 From: cournape at gmail.com (David Cournapeau) Date: Mon, 1 Oct 2012 19:33:05 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: On Mon, Oct 1, 2012 at 5:06 PM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> The code that produces yaml files. The point is precisely that it >> would be easy for me to consume this and produce the internal package >> representation in bento, which would then allow to configure, build >> and install packages using the bento format. > > Well, it's not pretty, but here's a Gist of the method which produces YAML > metadata from what's passed to setup(): > > https://gist.github.com/3812561 Thanks. I don't think it is fair to expect pretty code there in any case. I noticed that you put the classifiers list as a string (same for platform). I think it is expected to be a list, no ? Maybe slightly more controversial, I think the manifest should be "evaluated". The current system of inclusion + exclusion is too baroque to my taste, and makes it near-impossible to make reproducible builds. David From vinay_sajip at yahoo.co.uk Mon Oct 1 21:22:56 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 1 Oct 2012 19:22:56 +0000 (UTC) Subject: [Distutils] [Python-Dev] distlib updated with resources API References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: David Cournapeau gmail.com> writes: > I noticed that you put the classifiers list as a string (same for > platform). I think it is expected to be a list, no ? That's an oversight; there are doubtless others, too. > Maybe slightly more controversial, I think the manifest should be > "evaluated". The current system of inclusion + exclusion is too > baroque to my taste, and makes it near-impossible to make reproducible > builds. Would you care to give a little more detail about what you mean by "evaluate"? I've kept the manifest as it is for backward compatibility (i.e. so that my sanity checking of sdist follows the logic as is used by distutils/distribute). Regards, Vinay Sajip From lele at metapensiero.it Tue Oct 2 17:57:29 2012 From: lele at metapensiero.it (Lele Gaifax) Date: Tue, 02 Oct 2012 17:57:29 +0200 Subject: [Distutils] Struggling with a modular buildout configuration References: <87lihgp6gn.fsf@metapensiero.it> <87mx1rghmv.fsf@metapensiero.it> <87sjawbxp3.fsf@metapensiero.it> <877gs2jag1.fsf@metapensiero.it> Message-ID: <87k3v8vpcm.fsf@metapensiero.it> Hi, I just pushed an even simpler test case that shows the "defect": https://github.com/lelit/buildout/commit/1dc3d6af0b6326500ad72798352dd1537364b5ac that hopefully will help spotting the bug (or maybe my bad usage of the extends functionality :-) I already tried to tweak the _update_section() function, but without success: AFAIU, with the test above at some point the destination section (`s1` in the function) contains *both* a "egg" and a "egg +" keys, so the line # Find v1 in s2 first; it may have been defined locally too. v1, note1 = s2.get(key, s1.get(key, ("", ""))) where `key` is "egg" ignores the s1['egg +'] value... I'll try to spend some more hours on it. Thanks and bye, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From dholth at gmail.com Tue Oct 2 20:04:23 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 2 Oct 2012 14:04:23 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: <20121001121820.478adabf@limelight.wooz.org> References: <20121001104841.04693de5@limelight.wooz.org> <20121001121820.478adabf@limelight.wooz.org> Message-ID: On Mon, Oct 1, 2012 at 12:18 PM, Barry Warsaw wrote: > On Oct 01, 2012, at 03:43 PM, Vinay Sajip wrote: >>As a matter of interest, what are the APIs you're using which need filenames >>but can't use bytes or streams? dlopen() is the canonical example of an API that can only use a filename. From vinay_sajip at yahoo.co.uk Tue Oct 2 20:44:37 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 2 Oct 2012 18:44:37 +0000 (UTC) Subject: [Distutils] =?utf-8?q?pkg=5Fresources_API_use_according_to_Nulleg?= =?utf-8?q?e_search=09engine?= References: <20121001104841.04693de5@limelight.wooz.org> <20121001121820.478adabf@limelight.wooz.org> Message-ID: Daniel Holth gmail.com> writes: > dlopen() is the canonical example of an API that can only use a filename. Okay. Naturally there is already support for absolute paths in the file system for resources which are in the file system, so the question was really for resources in zips. Is it expected that the scenario will be quite common to get a .so or similar out of a package in a .zip into a cache, so that the name in the cache can be passed to dlopen()? Do you know of any specific PyPI distributions which do that, so I can look at them for testing etc? Regards, Vinay Sajip From pje at telecommunity.com Tue Oct 2 21:05:53 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 2 Oct 2012 15:05:53 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: References: <20121001104841.04693de5@limelight.wooz.org> <20121001121820.478adabf@limelight.wooz.org> Message-ID: On Tue, Oct 2, 2012 at 2:44 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> dlopen() is the canonical example of an API that can only use a filename. > > Okay. Naturally there is already support for absolute paths in the file system > for resources which are in the file system, so the question was really for > resources in zips. Is it expected that the scenario will be quite common to get > a .so or similar out of a package in a .zip into a cache, so that the name in the > cache can be passed to dlopen()? Do you know of any specific PyPI distributions > which do that, so I can look at them for testing etc? Just install any .egg distribution containing a C extension, without unzipping it. (bdist_egg builds eggs with the support for this inside them, using pkg_resources' API.) Btw, I think you might be missing part of the point of resource_filename() - it's not just that there are APIs (and external programs) that need real filenames, but that there are also C APIs that operate on OS-level file handles, so just having the bytes or a Python stream isn't sufficient. pkg_resources mainly included the support for this to deal with zipped C extensions, but also to support migrating code that still uses filenames internally for whatever reason, or needs them to pass to command line tools or C APIs. There's little point in making every application develop its own configuration and implementation for file extraction. From barry at python.org Tue Oct 2 21:17:20 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 2 Oct 2012 15:17:20 -0400 Subject: [Distutils] pkg_resources API use according to Nullege search engine In-Reply-To: References: <20121001104841.04693de5@limelight.wooz.org> <20121001121820.478adabf@limelight.wooz.org> Message-ID: <20121002151720.03772ca3@limelight.wooz.org> On Oct 02, 2012, at 06:44 PM, Vinay Sajip wrote: >Okay. Naturally there is already support for absolute paths in the file >system for resources which are in the file system, so the question was really >for resources in zips. Is it expected that the scenario will be quite common >to get a .so or similar out of a package in a .zip into a cache, so that the >name in the cache can be passed to dlopen()? Do you know of any specific PyPI >distributions which do that, so I can look at them for testing etc? This is kind of missing the point. You want to be able to write code against a single API and not care whether the file you need is already unpacked or in a zip. Cheers, -Barry From vinay_sajip at yahoo.co.uk Tue Oct 2 22:20:48 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 2 Oct 2012 20:20:48 +0000 (UTC) Subject: [Distutils] =?utf-8?q?pkg=5Fresources_API_use_according_to_Nulleg?= =?utf-8?q?e_search=09engine?= References: <20121001104841.04693de5@limelight.wooz.org> <20121001121820.478adabf@limelight.wooz.org> Message-ID: PJ Eby telecommunity.com> writes: > Btw, I think you might be missing part of the point of > resource_filename() - it's not just that there are APIs (and external > programs) that need real filenames, but that there are also C APIs > that operate on OS-level file handles, so just having the bytes or a > Python stream isn't sufficient. I think I was missing it - thanks to all of you who chipped in to clarify it for me. Regards, Vinay Sajip From p.f.moore at gmail.com Thu Oct 4 14:55:18 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 4 Oct 2012 13:55:18 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> Message-ID: On 28 September 2012 18:05, Nick Coghlan wrote: > On Fri, Sep 28, 2012 at 10:07 AM, Daniel Holth wrote: >> Are we trying to kill setuptools? I'm not entirely sure, but we should >> stop trying to do that. The migration should take essentially forever >> as soon as it makes sense for each pypi publisher. > > I'd certainly like to kill easy_install, and see any popular elements > of setuptools metadata become officially defined *independently* of > any given implementation. What I would like to see is: 1. Every packaging tool creating standards-based (dist-info format) metadata files. That includes distutils (the "feature freeze" notwithstanding) as that would catch many existing packages. I'd also like to see distribute switch to using dist-info rather than egg-info, although I'm not sure if that's a separate thing or if distribute just reuses distutils here. New tools like Bento should be using the dist-info format. Tools that consume metadata (e.g., pip install/pip freeze) could then focus on the dist-info format, retaining any other support as legacy-only. 2. Some level of standardised functionality for building and installing. By "standardised", I mean that given *any* sdist, no matter what build tool it uses under the scenes, there is a way of saying "build this, and put the output into the following directories". For distutils, this is --install-XXX. Distribute complicates this by changing the meaning of these options (you need --single-version-externally-managed and --record) but otherwise follows the standard. Add an API to wrap this (include autodetection of setuptools/distribute, and hooks for other tools to integrate in, and you're done). A wrapper like this might be a good thing to have in distlib. 2a. A common, introspectable, format for describing a distribution and its build Not necessary for the above, but useful separately to allow tools to check whether they need to install dependencies before building, for example. I don't see a way of having this affect existing packages, though, short of some form of setup.py converter. So it'll only be viable to depend on this when a new build tool has established itself. 3. A standard layout for installed files. This gets harder, because OS conventions come into play. But basically, the sysconfig locations are the way to encapsulate this. Oh, and kill the egg format :-) (Seriously, does the egg format offer any benefit other than the old multiversion support that no-one uses any more? If not, it probably should be allowed to die off). 4. A standard binary install format (wheel! :-)) 5. Conversion tools to build things like RPMs or MSIs from a wheel would likely be the best way to integrate platform-format installers. The other aspect of easy_install is the package location bit. I believe Vinay has added something like this (the PyPI search and web scraping code) to distlib. Paul. From cournape at gmail.com Thu Oct 4 15:13:03 2012 From: cournape at gmail.com (David Cournapeau) Date: Thu, 4 Oct 2012 14:13:03 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> <472D350E502F4B2FA46F3B6475281BE6@gmail.com> <14DC3D7670F4434E970EDD724F1D7660@gmail.com> Message-ID: On Mon, Oct 1, 2012 at 8:22 PM, Vinay Sajip wrote: > David Cournapeau gmail.com> writes: > >> I noticed that you put the classifiers list as a string (same for >> platform). I think it is expected to be a list, no ? > > That's an oversight; there are doubtless others, too. Sure. I guess I was just trying to get at getting the code released in a repo so that we can provide patches :) > >> Maybe slightly more controversial, I think the manifest should be >> "evaluated". The current system of inclusion + exclusion is too >> baroque to my taste, and makes it near-impossible to make reproducible >> builds. > > Would you care to give a little more detail about what you mean by > "evaluate"? I've kept the manifest as it is for backward compatibility (i.e. > so that my sanity checking of sdist follows the logic as is used by > distutils/distribute). If you want to be backward compatible with distutils, then yes, you have to encode it as is. By evaluating, I meant specifying the list of files instead of using some higher level logic. Otherwise, the static format does not specify the actual content (and depends on way too many parameters). David From dholth at gmail.com Thu Oct 4 17:20:36 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Oct 2012 11:20:36 -0400 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> Message-ID: On Thu, Oct 4, 2012 at 8:55 AM, Paul Moore wrote: > On 28 September 2012 18:05, Nick Coghlan wrote: >> On Fri, Sep 28, 2012 at 10:07 AM, Daniel Holth wrote: >>> Are we trying to kill setuptools? I'm not entirely sure, but we should >>> stop trying to do that. The migration should take essentially forever >>> as soon as it makes sense for each pypi publisher. >> >> I'd certainly like to kill easy_install, and see any popular elements >> of setuptools metadata become officially defined *independently* of >> any given implementation. > 3. A standard layout for installed files. This gets harder, because OS > conventions come into play. But basically, the sysconfig locations are > the way to encapsulate this. Oh, and kill the egg format :-) > (Seriously, does the egg format offer any benefit other than the old > multiversion support that no-one uses any more? If not, it probably > should be allowed to die off). > > 4. A standard binary install format (wheel! :-)) I envision a system that uses a PEP site-packages-directory-with-one-distribution as the interface between build and install. The build system always installs into an empty directory with a predictable layout: module.py package/__init__.py distname-1.0.data/scripts/a-script.py distname-1.0.dist-info/METADATA The build system does not have to worry about annoying details like counting the number of packages that use a particular __init__.py or rename-installed-configuration-file-to-.orig only if it was modified from the previous install. You don't have to give the build system elevated privileges because it does not install. Once the build system is done, the installer knows how to copy the dist to its destination paths (potentially running as root, or on another machine that does not have a compiler) instead of necessarily putting it all under a single path. The installer worries about not overwriting config files and so forth. When the interface between build and install is simple "build a standard thing" -> "install a standard thing" it's not a problem to implement features like "index the metadata with sqlite" that you could never put in all the build systems. The other huge advantage to all the setuptools and distutils haters out there is that you no longer need to have the build system installed in the target environment, so you can use setuptools to build without having to install it to run. The egg format is technically equivalent to a site packages directory with only one thing in it. The only thing we change is to rename EGG-INFO to distname-1.0.dist-info and rename the searchable name PKG-INFO to the un-googleable name METADATA, and put more of the metadata into one file METADATA. Nothing is lost, so people who need to use lots of site-packages directories can still do so. Maybe those people will name them distname-1.0.dist/ or like gem put them in a folder where every subfolder is a dist. I don't see a reason to violently kill eggs as long as it is very easy to avoid using them. Instead of a RESOURCES file with all the files in a dist listed line-by-line (slow) the installer should be able to write the list of installation prefixes to a file distribution/_install_locations.py (the name is configurable, and it is only written on request). This idea comes from Bento. purelib = "../relative/path/to-purelib" platlib = "../relative/path/to-platlib" IIRC distutils in the standard library only knows how to generate PKG-INFO at the root of an sdist and it never does anything with the metadata. It would not be a big deal to copy that into a .dist-info directory. I would support a subset of the pysetup setup.cfg [metadata] for setup- and install- requirements instead of adding them to setup.py. From ncoghlan at gmail.com Thu Oct 4 18:47:55 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 4 Oct 2012 22:17:55 +0530 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> Message-ID: On Thu, Oct 4, 2012 at 6:25 PM, Paul Moore wrote: > 2. Some level of standardised functionality for building and > installing. By "standardised", I mean that given *any* sdist, no > matter what build tool it uses under the scenes, there is a way of > saying "build this, and put the output into the following > directories". For distutils, this is --install-XXX. Distribute > complicates this by changing the meaning of these options (you need > --single-version-externally-managed and --record) but otherwise > follows the standard. Add an API to wrap this (include autodetection > of setuptools/distribute, and hooks for other tools to integrate in, > and you're done). A wrapper like this might be a good thing to have in > distlib. Note that one of the core goals of wheel is to finally break the distutils conflation of "build" and "install". They're different things, but the lack of a platform-neutral binary format has meant that they've largely been conflated in practice for Python distribution. Any new build hook should be as hands off as possible and have a wheel (or a wheel-format directory) as the result, which is then installed according to the normal rules. > 5. Conversion tools to build things like RPMs or MSIs from a wheel > would likely be the best way to integrate platform-format installers. I actually need to be able to generate an SRPM from an sdist, as SRPM -> RPM is the only build system Koji understands. However, the advantage of wheel is that it should be possible to automatically generate a SPEC file which is actually half decent (by running the build hook and introspecting the resulting wheel). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From cournape at gmail.com Thu Oct 4 18:56:20 2012 From: cournape at gmail.com (David Cournapeau) Date: Thu, 4 Oct 2012 17:56:20 +0100 Subject: [Distutils] [Python-Dev] distlib updated with resources API In-Reply-To: References: <50636845.8080204@ziade.org> <50645C34.6070606@ziade.org> <1348756086.3495.1.camel@localhost.localdomain> <50646CFF.3010601@ziade.org> <50647599.3030507@ziade.org> <5064E31D.80601@ziade.org> Message-ID: On Thu, Oct 4, 2012 at 5:47 PM, Nick Coghlan wrote: > On Thu, Oct 4, 2012 at 6:25 PM, Paul Moore wrote: >> 2. Some level of standardised functionality for building and >> installing. By "standardised", I mean that given *any* sdist, no >> matter what build tool it uses under the scenes, there is a way of >> saying "build this, and put the output into the following >> directories". For distutils, this is --install-XXX. Distribute >> complicates this by changing the meaning of these options (you need >> --single-version-externally-managed and --record) but otherwise >> follows the standard. Add an API to wrap this (include autodetection >> of setuptools/distribute, and hooks for other tools to integrate in, >> and you're done). A wrapper like this might be a good thing to have in >> distlib. > > Note that one of the core goals of wheel is to finally break the > distutils conflation of "build" and "install". They're different > things, but the lack of a platform-neutral binary format has meant > that they've largely been conflated in practice for Python > distribution. Any new build hook should be as hands off as possible > and have a wheel (or a wheel-format directory) as the result, which is > then installed according to the normal rules. > >> 5. Conversion tools to build things like RPMs or MSIs from a wheel >> would likely be the best way to integrate platform-format installers. > > I actually need to be able to generate an SRPM from an sdist, as SRPM > -> RPM is the only build system Koji understands. However, the > advantage of wheel is that it should be possible to automatically > generate a SPEC file which is actually half decent (by running the > build hook and introspecting the resulting wheel). Note that the build manifest concept as used in bento (and used by wheel as well I believe) is designed specifically to allow conversion between binary formats (where it makes sense of course). It is still in flux, but I could convert eggs to windows installers, David From vinay_sajip at yahoo.co.uk Fri Oct 5 00:28:01 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 4 Oct 2012 22:28:01 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought Message-ID: I've updated distlib as follows: 1. I've added support for the equivalent of "pkg_resources.resource_filename" for returning a true filesystem path for a resource. I've added info about this in the "Distlib's design" document, though the reference API has not been fully updated - I thought I'd see if there was any feedback about the approach I've used. Code and tests have been added, though. 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only tweaked it a little to run under 2.x and 3.x from a single code base, ensuring that all the tests passed. I hope to update it with better support for finding archives not hosted on PyPI, using hints. I'm also considering the following changes, which I'd like some feedback on: * Merging the distlib.depgraph package into the distlib.database package, as they are fairly interlinked (e.g. adding requires() and required_by() methods to Distribution). * Removing depgraph.main() and its test, because I don't think it belongs in the library layer, but is better implemented as a utility script. Please let me have your comments. Regards, Vinay Sajip From dholth at gmail.com Fri Oct 5 05:15:10 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 4 Oct 2012 23:15:10 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: Message-ID: On Thu, Oct 4, 2012 at 6:28 PM, Vinay Sajip wrote: > I've updated distlib as follows: > > 1. I've added support for the equivalent of "pkg_resources.resource_filename" > for returning a true filesystem path for a resource. I've added info about > this in the "Distlib's design" document, though the reference API has not > been fully updated - I thought I'd see if there was any feedback about the > approach I've used. Code and tests have been added, though. :-) I still like the "mimic os.listdir but with an additional package argument" API because it has fewer parts. Is there a reason why we can't just fix pkgutil? It is probably much easier to do it in distlib. https://bitbucket.org/dholth/distlib2/src/tip/_distlib2/resources.py implements the distlib resources API using pkg_resources as the implementation. Check out how neat apipkg is in https://bitbucket.org/dholth/distlib2/src/tip/distlib/__init__.py. distlib2 does yet not implement any of the other distlib APIs. > 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only > tweaked it a little to run under 2.x and 3.x from a single code base, > ensuring that all the tests passed. I hope to update it with better support > for finding archives not hosted on PyPI, using hints. +0 > I'm also considering the following changes, which I'd like some feedback on: > > * Merging the distlib.depgraph package into the distlib.database package, as > they are fairly interlinked (e.g. adding requires() and required_by() methods > to Distribution). +1 obviously since I asked > * Removing depgraph.main() and its test, because I don't think it belongs in the > library layer, but is better implemented as a utility script. +1 an installer probably only needs the toposort of the dependency graph From tarek at ziade.org Fri Oct 5 09:40:52 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 05 Oct 2012 09:40:52 +0200 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: Message-ID: <506E8F04.5060109@ziade.org> On 10/5/12 12:28 AM, Vinay Sajip wrote: > I've updated distlib as follows: > > 1. I've added support for the equivalent of "pkg_resources.resource_filename" > for returning a true filesystem path for a resource. I've added info about > this in the "Distlib's design" document, though the reference API has not > been fully updated - I thought I'd see if there was any feedback about the > approach I've used. Code and tests have been added, though. > > 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only > tweaked it a little to run under 2.x and 3.x from a single code base, > ensuring that all the tests passed. I hope to update it with better support > for finding archives not hosted on PyPI, using hints. > > I'm also considering the following changes, which I'd like some feedback on: > > * Merging the distlib.depgraph package into the distlib.database package, as > they are fairly interlinked (e.g. adding requires() and required_by() methods > to Distribution). > > * Removing depgraph.main() and its test, because I don't think it belongs in the > library layer, but is better implemented as a utility script. I like the idea of having a main section to play with the tools, people can invoke with -m I don't think its a bad idea to have it in each module, like what Python does for SimpleHTTPServer or json.tools stuff we could have: - a manifest reader with various options - displaying a dependency graph - querying some stuff at PyPI.. etc.. > > Please let me have your comments. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Fri Oct 5 11:32:16 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 09:32:16 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: Message-ID: Daniel Holth gmail.com> writes: > I still like the "mimic os.listdir but with an additional package > argument" API because it has fewer parts. Is there a reason why we > can't just fix pkgutil? It is probably much easier to do it in > distlib. Sorry, I didn't get what you meant in the above. Please clarify. Thanks for the feedback. Regards, Vinay Sajip From p.f.moore at gmail.com Fri Oct 5 12:10:44 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 11:10:44 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 5 October 2012 10:27, Vinay Sajip wrote: >> +1 on this. Can you share a little more on your hint idea? I have a > >> specific use case in mind, and would like to know if it's covered. >> It's an intranet webpage that hosts packages in a very odd format, >> unfortunately, so I need to write code to get the packages, just >> providing URLs isn't possible. > > For example, setuptools uses "dependency_links" which indicate URLs which lead to downloadable packages. I haven't thought it through in detail, but my package.yaml format preserves those links. I would aim to design in such a way that custom schemes aren't hard to support. One more area I need to look at is pip's PackageFinder, which is the minimum standard we need to support. [Sorry, I hadn't meant to go off-list - blame gmail...] Right, that's not going to be enough for me. I have a web page where files are hosted at http://my.domain/datastore/VARYING_ENCODED_ID/filename.tar.gz. The problem is that VARYING_ENCODED_ID changes regularly. It's possible to calculate it in code, but only based on the current date and the package name :-( It really isn't worth asking why, but I'm stuck with it. I'd envisage a design consisting of independent "Locators", one for a local directory, one for PyPI using XMLRPC, one for PyPI's "simple" webpage interface (IMO, we need both PyPI types, as the simple interface generates a lot of useless web scraping - look at the simple page for lxml, for example...). Users could add locators for any custom stores as needed. I wrote a 30-minute proof of concept, which is at https://bitbucket.org/pmoore/tools/src/67b33c15efad/bin/Locator.py if you're interested. Plan B for something like this would be to use one of the "run a local PyPI server" packages in existence, and write a front-end to the web store that I can run locally, and point to as an extra PyPI index. That may well be a more general solution, but I don't think any of the existing PyPI server packages supports this type of usage, so it would need some work. Paul. From vinay_sajip at yahoo.co.uk Fri Oct 5 12:19:59 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 10:19:59 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <506E8F04.5060109@ziade.org> Message-ID: Tarek Ziad? ziade.org> writes: > I like the idea of having a main section to play with the tools, people > can invoke with -m > > I don't think its a bad idea to have it in each module, like what Python > does > for SimpleHTTPServer or json.tools > I've no problem with having a "if __name == '__main__'" section, but I don't think it should need to be unit tested, or form part of the contract of the module. For example, the depgraph.main() test as it stands is dependent on whatever happens to be on sys.path on the machine running the test, possibly leading to results unreproducible by different people. For example, on my machine, the test originally failed because of version numbers of legacy distributions on sys.path, which didn't meet PEP 386 standards and led to exceptions being raised in depgraph.main(). I commented out the "raise" and replaced it with a "continue", and the test then passed. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Oct 5 12:29:00 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 10:29:00 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > Right, that's not going to be enough for me. I have a web page where > files are hosted at > http://my.domain/datastore/VARYING_ENCODED_ID/filename.tar.gz. The > problem is that VARYING_ENCODED_ID changes regularly. It's possible to > calculate it in code, but only based on the current date and the > package name It really isn't worth asking why, but I'm stuck with > it. > > I'd envisage a design consisting of independent "Locators", one for a > local directory, one for PyPI using XMLRPC, one for PyPI's "simple" > webpage interface (IMO, we need both PyPI types, as the simple > interface generates a lot of useless web scraping - look at the simple > page for lxml, for example...). Users could add locators for any > custom stores as needed. I wrote a 30-minute proof of concept, which > is at https://bitbucket.org/pmoore/tools/src/67b33c15efad/bin/Locator.py > if you're interested. I'll take a look - it sounds like the sort of approach I would take, to have a high level interface for locating distributions that allows enumeration of distributions, means of fetching them etc., with the ability to slot in implementations for unusual cases like yours. Regards, Vinay Sajip From dholth at gmail.com Fri Oct 5 13:00:54 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 07:00:54 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: Message-ID: <8B7D8A8D-EE0B-41F7-AF61-0998746E3AC1@gmail.com> I mean should there be a pkgutil.pkg_listdir, etc. to accompany pkgutil.get_data Daniel Holth On Oct 5, 2012, at 5:32 AM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> I still like the "mimic os.listdir but with an additional package >> argument" API because it has fewer parts. Is there a reason why we >> can't just fix pkgutil? It is probably much easier to do it in >> distlib. > > Sorry, I didn't get what you meant in the above. Please clarify. > > Thanks for the feedback. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From tarek at ziade.org Fri Oct 5 14:40:21 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Fri, 05 Oct 2012 14:40:21 +0200 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <506E8F04.5060109@ziade.org> Message-ID: <506ED535.7060004@ziade.org> On 10/5/12 12:19 PM, Vinay Sajip wrote: > Tarek Ziad? ziade.org> writes: > >> I like the idea of having a main section to play with the tools, people >> can invoke with -m >> >> I don't think its a bad idea to have it in each module, like what Python >> does >> for SimpleHTTPServer or json.tools >> > I've no problem with having a "if __name == '__main__'" section, but I don't > think it should need to be unit tested, or form part of the contract of the > module. oh yeah - I don't care about this part :) From dholth at gmail.com Fri Oct 5 15:01:17 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 09:01:17 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Check out the reasonably modular and cleanly written pyg installer. https://github.com/rubik/pyg/ It has its own independent package database / metadata parser. https://github.com/rubik/pkgtools/blob/master/pkgtools/pkg.py It has a pypi interface too. From dholth at gmail.com Fri Oct 5 15:47:14 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 09:47:14 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 6:29 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> Right, that's not going to be enough for me. I have a web page where >> files are hosted at >> http://my.domain/datastore/VARYING_ENCODED_ID/filename.tar.gz. The >> problem is that VARYING_ENCODED_ID changes regularly. It's possible to >> calculate it in code, but only based on the current date and the >> package name It really isn't worth asking why, but I'm stuck with >> it. >> >> I'd envisage a design consisting of independent "Locators", one for a >> local directory, one for PyPI using XMLRPC, one for PyPI's "simple" >> webpage interface (IMO, we need both PyPI types, as the simple >> interface generates a lot of useless web scraping - look at the simple >> page for lxml, for example...). Users could add locators for any >> custom stores as needed. I wrote a 30-minute proof of concept, which >> is at https://bitbucket.org/pmoore/tools/src/67b33c15efad/bin/Locator.py >> if you're interested. > > I'll take a look - it sounds like the sort of approach I would take, to have a > high level interface for locating distributions that allows enumeration of > distributions, means of fetching them etc., with the ability to slot > in implementations for unusual cases like yours. Is this supposed to go into the stdlib? Simple reference implementations (wsgiref) fare better there than frameworks (distutils). The pluggable installer metaframework belongs on pypi. From vinay_sajip at yahoo.co.uk Fri Oct 5 16:13:41 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 14:13:41 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Daniel Holth gmail.com> writes: > Is this supposed to go into the stdlib? Simple reference > implementations (wsgiref) fare better there than frameworks > (distutils). The pluggable installer metaframework belongs on pypi. I'm thinking of distlib as a library rather than a framework. However, it's probably too early to say definitively which parts might belong in the stdlib and which mightn't. Not every library with a service provider interface is automatically a framework. Regards, Vinay Sajip From p.f.moore at gmail.com Fri Oct 5 16:23:25 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 15:23:25 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 5 October 2012 14:47, Daniel Holth wrote: > Is this supposed to go into the stdlib? Simple reference > implementations (wsgiref) fare better there than frameworks > (distutils). The pluggable installer metaframework belongs on pypi. I disagree. Having an installer depend on external packages is a practical problem. And having every installer invent (or not) its own means of allowing users to add custom package repositories is also an issue. Having a basic implementation supporting the PSF-supported repository (PyPI) but including simple hooks to allow users to add extra ones gives the benefit of a reference implementation as well as encouraging by example the provision of flexibility. No-one could try to claim that the sort of web-scraping that easy_install/pip does is a "simple" reference implementation, either. If you take that viewpoint, I'd say the stdlib implementation should *only* use the XMLRPC interface to PyPI. Code to use the "simple" interface and trawl all those links looking for distribution files can't be justified in the stdlib for any *other* reason than to save anyone else ever having to write it again :-) Paul. PS If you want to start over-engineering the flexibility, users should have a way of choosing whether to use the webscraper or XMLRPC interfaces to PyPI. The former finds more packages (as I understand it) whereas the latter is much faster. As someone who's never needed a package that can't be found using both interfaces (or neither :-() I deeply resent the speed penalties imposed by the "simple" interface (hence my repeated insistence on quoting the word "simple", as I find it far from simple :-)) PPS If my locator interface ever matures enough, I'm happy to release it on PyPI. But I don't want to compete with Vinay or a stdlib implementation, so I'd rather co-operate on a unified view of how to approach the problem. From vinay_sajip at yahoo.co.uk Fri Oct 5 16:37:29 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 14:37:29 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > No-one could try to claim that the sort of web-scraping that > easy_install/pip does is a "simple" reference implementation, either. > If you take that viewpoint, I'd say the stdlib implementation should > *only* use the XMLRPC interface to PyPI. Code to use the "simple" > interface and trawl all those links looking for distribution files > can't be justified in the stdlib for any *other* reason than to save > anyone else ever having to write it again [...] > PS If you want to start over-engineering the flexibility, users should > have a way of choosing whether to use the webscraper or XMLRPC > interfaces to PyPI. The former finds more packages (as I understand > it) whereas the latter is much faster. As someone who's never needed a > package that can't be found using both interfaces (or neither ) I Is that really the case? I'd assumed that the simple pages were generated from the package database created from uploads to PyPI, so I would have expected querying the XML-RPC interface to produce the same results as from scraping the HTML (allowing for the possibility that, if the HTML pages are generated periodically as static files from the database, they might be stale at times). I thought that pip needed to scrape pages because people host distribution archives on servers other than PyPI (e.g. Google code, GitHub, BitBucket or their own servers), with the links to those archives navigable through e.g. the "dependency_links" argument to setup(), or the URLs mentioned in the PyPI metadata. Regards, Vinay Sajip From p.f.moore at gmail.com Fri Oct 5 17:04:30 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 16:04:30 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 5 October 2012 15:37, Vinay Sajip wrote: >> PS If you want to start over-engineering the flexibility, users should >> have a way of choosing whether to use the webscraper or XMLRPC >> interfaces to PyPI. The former finds more packages (as I understand >> it) whereas the latter is much faster. As someone who's never needed a >> package that can't be found using both interfaces (or neither ) I > > Is that really the case? I'd assumed that the simple pages were generated from > the package database created from uploads to PyPI, so I would have expected > querying the XML-RPC interface to produce the same results as from scraping the > HTML (allowing for the possibility that, if the HTML pages are generated > periodically as static files from the database, they might be stale at times). Well, yes. But the static files don't make it easy to distinguish the different categories they contain (see below) > I thought that pip needed to scrape pages because people host distribution > archives on servers other than PyPI (e.g. Google code, GitHub, BitBucket or > their own servers), with the links to those archives navigable through e.g. the > "dependency_links" argument to setup(), or the URLs mentioned in the PyPI > metadata. I don't know how true that is these days - I don't think I've ever personally encountered a package that wasn't either available from the PyPI download URL (release_urls() in the XMLRPC interface) or unavailable via pip. But my range of packages tried is fairly limited... The static pages merge all of the following information: 1. The download URLs you can get from the XMLRPC interface release_urls, but with all releases covered in a single place. 2. release_data[download_url] which is available from the XMLRPC interface 3. Other URLs from release_data (home_page, project_url). The first ones are fine, as they point to files. The second is often a file, and seems to frequently duplicate the first. I'm not sure how useful it is. The final one often points to a further webpage - I presume that's what you plan to scrape. That's where the issue lies, though, as at least some of those links time out (lxml's does, IIRC) and as I say, I don't think I know of a case where it's actually worth doing. But this is based on a very superficial and limited experience. I'll happily bow to better information. On the other hand, is manually parsing the static page any faster in a practical sense than using XMLRPC? Paul. From sklein at cpcug.org Fri Oct 5 16:54:24 2012 From: sklein at cpcug.org (Stanley A. Klein) Date: Fri, 5 Oct 2012 10:54:24 -0400 (EDT) Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: Message-ID: <51528.71.163.193.12.1349448864.squirrel@www.cpcug.org> I looked at the documentation and can't completely follow what you are doing, although I was told in another email that the issue I raised was being addressed in pkg_resources. Here is what I need: I'm trying to build an rpm package from a Pypi tar.gz. The tar file contains several files and subdirectories. I need to map them as follows: a. Send the package modules to /usr/lib/pythonx.x/site-packages/name_of_pkg b. Send the executable command scripts to /usr/bin or /usr/sbin as appropriate c. Send any readme files, license files, doc directories, example directories, test directories, and other such files/directories to /usr/share/doc/name_of_pkg d. Generally, send the various files to places prescribed in the Filesystem Hierarchy Standard for Unix/Linux. I can't completely see how I would do this using what is described in the distlib documentation. Stan Klein On Fri, October 5, 2012 6:00 am, Vinay Sajip wrote: > Date: Thu, 4 Oct 2012 22:28:01 +0000 (UTC) > From: Vinay Sajip > To: Distutils-Sig at Python.Org > Subject: [Distutils] distlib updated - comments sought > > I've updated distlib as follows: > > 1. I've added support for the equivalent of > "pkg_resources.resource_filename" > for returning a true filesystem path for a resource. I've added info > about > this in the "Distlib's design" document, though the reference API has > not > been fully updated - I thought I'd see if there was any feedback about > the > approach I've used. Code and tests have been added, though. > > 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only > tweaked it a little to run under 2.x and 3.x from a single code base, > ensuring that all the tests passed. I hope to update it with better > support > for finding archives not hosted on PyPI, using hints. > > I'm also considering the following changes, which I'd like some feedback > on: > > * Merging the distlib.depgraph package into the distlib.database package, > as > they are fairly interlinked (e.g. adding requires() and required_by() > methods > to Distribution). > > * Removing depgraph.main() and its test, because I don't think it belongs > in the > library layer, but is better implemented as a utility script. > > Please let me have your comments. > > Regards, > > Vinay Sajip > From vinay_sajip at yahoo.co.uk Fri Oct 5 17:39:09 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 15:39:09 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > The first ones are fine, as they point to files. The second is often a > file, and seems to frequently duplicate the first. I'm not sure how > useful it is. The final one often points to a further webpage - I > presume that's what you plan to scrape. That's where the issue lies, > though, as at least some of those links time out (lxml's does, IIRC) > and as I say, I don't think I know of a case where it's actually worth > doing. > > But this is based on a very superficial and limited experience. I'll > happily bow to better information. > > On the other hand, is manually parsing the static page any faster in a > practical sense than using XMLRPC? Well, XML-RPC is of course preferable; the current code in distlib is just whatever I copied across from packaging, but the next step will be to look at the releases which are available from the different sources (XML-RPC, PyPI metadata URLs, dependency_links etc.) to see what sorts of things wouldn't be accessible if we restricted to say, just using XML-RPC. Since all the information in the static pages seems to be available via XML-RPC, what is the point of the simple interface, other than for occasional viewing by a human? Regards, Vinay Sajip From dholth at gmail.com Fri Oct 5 17:40:50 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 11:40:50 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 11:39 AM, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> The first ones are fine, as they point to files. The second is often a >> file, and seems to frequently duplicate the first. I'm not sure how >> useful it is. The final one often points to a further webpage - I >> presume that's what you plan to scrape. That's where the issue lies, >> though, as at least some of those links time out (lxml's does, IIRC) >> and as I say, I don't think I know of a case where it's actually worth >> doing. >> >> But this is based on a very superficial and limited experience. I'll >> happily bow to better information. >> >> On the other hand, is manually parsing the static page any faster in a >> practical sense than using XMLRPC? > > Well, XML-RPC is of course preferable; the current code in distlib is just > whatever I copied across from packaging, but the next step will be to look > at the releases which are available from the different sources (XML-RPC, > PyPI metadata URLs, dependency_links etc.) to see what sorts of things wouldn't > be accessible if we restricted to say, just using XML-RPC. Since all the > information in the static pages seems to be available via XML-RPC, what is the > point of the simple interface, other than for occasional viewing by a human? IIRC the most practical limitation is that the XML-RPC interface doesn't exist on the mirrors. From vinay_sajip at yahoo.co.uk Fri Oct 5 17:49:04 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 15:49:04 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <51528.71.163.193.12.1349448864.squirrel@www.cpcug.org> Message-ID: Stanley A. Klein cpcug.org> writes: > I looked at the documentation and can't completely follow what you are > doing, although I was told in another email that the issue I raised was > being addressed in pkg_resources. The goal of distlib is to provide a library of low-level functions relating to packaging and distribution of Python software, and it does not contain any installers (though it will contain low-level code which could be used by an installer). The distlib code aims to serve as an implementation of the various packaging PEPs, such that if higher level tools use distlib, they should be able to both save writing code (and thereby gain time) as well as achieve better interoperability with other packaging tools. The goal of distlib is not to be a complete packaging solution (which packaging tried to be for Python 3.3, but it proved to be too ambitious a goal). Which issue are you referring to, which you were told was addressed by pkg_resources? From the section starting "Here is what I need", I couldn't see how pkg_resources could fulfil those needs by itself. > Here is what I need: [snip] > I can't completely see how I would do this using what is described in the > distlib documentation. What you need is an installer, and distlib doesn't contain functionality to do installation. I won't rehash the various discussions around the needs of Linux distro packagers, except to say that the PEPs have taken into account the needs of distro packagers and the ability to map various parts of a distribution to various target directories in a distro-specific manner. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Oct 5 17:52:48 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 15:52:48 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Daniel Holth gmail.com> writes: > IIRC the most practical limitation is that the XML-RPC interface > doesn't exist on the mirrors. Of course. It's easier to replicate a static website than an XML-RPC server to Web Scale ;-), but I presume it's only a question of hosting resources and developer time to make mirrors which are XML-RPC capable? Regards, Vinay Sajip From dholth at gmail.com Fri Oct 5 18:04:07 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 12:04:07 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 10:23 AM, Paul Moore wrote: > On 5 October 2012 14:47, Daniel Holth wrote: > PPS If my locator interface ever matures enough, I'm happy to release > it on PyPI. But I don't want to compete with Vinay or a stdlib > implementation, so I'd rather co-operate on a unified view of how to > approach the problem. "The Problem" Bootstrapping is kinda annoying because Python doesn't include an installer for pip or buildout or ... and it can be hard to choose between the many excellent installers that are available on and off of pypi. ~1300 of the ~20000 packages on pypi have trouble using setup.py as their build system / metadata source format. For the ~1300 broken packages, distutils is awful because it is not really extensible, though setuptools tried. People have to install setuptools against their will because there is only one implementation of the pkg_resources API and 75% of the packages on pypi require setuptools. Packaging has been in turmoil for years waiting for something. In my estimation we're not saving the world here. The goal should be to fix 1,300 packages without breaking 19,000, to make bootstrapping easier, and to make setuptools optional but neither required nor prohibited. It is wonderful to have distlib. I support it. I'm playing with the competing distlib2 implementation so that both APIs can be better, so we can find out which parts provide functionality that does not just have a different name in pkg_resources, and so that it can be possible to replace the implementation without changing the API. If your goal is to avoid "implementation defined behavior" it's a good idea to have two. From p.f.moore at gmail.com Fri Oct 5 19:24:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 18:24:11 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 5 October 2012 17:04, Daniel Holth wrote: > ~1300 of the ~20000 packages on pypi have trouble using setup.py as > their build system / metadata source format. That's interesting information. Do you know in what way they have trouble with setup.py? Do they not use it at all, do they need features it doesn't provide, or what? > For the ~1300 broken packages, distutils is awful because it is not > really extensible, though setuptools tried. Yeah, that's the common complaint. Plus, "it's too extensible" :-) (From people trying to change it who have to deal with all the fancy hacks people have done). > People have to install setuptools against their will because there is > only one implementation of the pkg_resources API and 75% of the > packages on pypi require setuptools. I wish we could separate pkg_resources and setuptools. I'd love to know which packages needed each (but I suspect that's not a question that can be answered without looking at the actual code). Ignoring the egg support aspects, pkg_resources is something that could be replaced - a reasonable proportion of distlib offers alternatives to the pkg_resources code, and more could be added. On the other hand, setuptools per se is almost entirely a build time facility, so shouldn't be needed at runtime (and so using it for build should be relatively unimportant). Paul From dholth at gmail.com Fri Oct 5 19:49:51 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 13:49:51 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 1:24 PM, Paul Moore wrote: > On 5 October 2012 17:04, Daniel Holth wrote: > >> ~1300 of the ~20000 packages on pypi have trouble using setup.py as >> their build system / metadata source format. > > That's interesting information. Do you know in what way they have > trouble with setup.py? Do they not use it at all, do they need > features it doesn't provide, or what? I'm basing this only on Vinay's numbers of how-many-packages-can-generate-a-yaml. It's probably mostly packages that import something he didn't have installed inside setup.py, but I don't have a good way to find out exactly what is wrong with each one. An awful lot of packages do work fine; I don't have an automated way to detect that either. >> People have to install setuptools against their will because there is >> only one implementation of the pkg_resources API and 75% of the >> packages on pypi require setuptools. > > I wish we could separate pkg_resources and setuptools. I'd love to > know which packages needed each (but I suspect that's not a question > that can be answered without looking at the actual code). Ignoring the > egg support aspects, pkg_resources is something that could be replaced > - a reasonable proportion of distlib offers alternatives to the > pkg_resources code, and more could be added. On the other hand, > setuptools per se is almost entirely a build time facility, so > shouldn't be needed at runtime (and so using it for build should be > relatively unimportant). And there is a good bit of pkg_resources.py that is only needed at install time, if you don't mind giving up [console_scripts] dependency loading. It would probably be a win to only populate working_set lazily at least. Whatever you did would probably mean the loss of a feature for some users, but that might be OK if they knew what to expect. From vinay_sajip at yahoo.co.uk Fri Oct 5 19:57:25 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 17:57:25 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Daniel Holth gmail.com> writes: > Bootstrapping is kinda annoying because Python doesn't include an > installer for pip or buildout or ... and it can be hard to choose > between the many excellent installers that are available on and off of > pypi. That was the point of packaging - to have something better than distutils in the stdlib. The original goal was too ambitious to achieve in the 3.3 timeframe, and perhaps issue could be taken with some use cases which weren't fully considered. Since packaging is an infrastructure concern, I believe (as others do) that there's a place for *something* in the stdlib that's better than distutils, because distutils failed to meet changing needs. That something may be distlib, or something like it, or nothing like it. It's too early to say. > ~1300 of the ~20000 packages on pypi have trouble using setup.py as > their build system / metadata source format. I'm not sure what you mean. Packages don't have trouble, people do. For example, it may be possible for me to install a particular package from PyPI on my system (=> "no trouble"), but the package may be hard to package for Linux distros, because some of the installation logic happens in setup.py or code called from it. > For the ~1300 broken packages, distutils is awful because it is not > really extensible, though setuptools tried. Valiant effort by setuptools, but it could be considered a band-aid. Obviously there are different opinions about setuptools, but it's hard to argue against the fact that setuptools and pkg_resources are not considered worthy of inclusion in the stdlib by python-dev. > People have to install setuptools against their will because there is > only one implementation of the pkg_resources API and 75% of the > packages on pypi require setuptools. Well, isn't that what packaging was (and distlib is) trying to remedy? > Packaging has been in turmoil for years waiting for something. Mainly, people with the time and inclination :-) > In my estimation we're not saving the world here. Just trying to improve the world a teeny little bit is a worthy goal; saving the world is beyond the reach of most of us. > The goal should be > to fix 1,300 packages without breaking 19,000, to make bootstrapping > easier, and to make setuptools optional but neither required nor > prohibited. I'm not sure that anyone is anticipating, or working towards, breaking thousands of packages. Nothing is "prohibited" - people can use whatever they want. But setuptools (or other third-party package) can't be truly optional while the stdlib lacks functionality which people need, and setuptools provides. So, perhaps the goal is just to offer more choices. People don't like change, but change can't always be avoided. We can be optimistic that strategies will be in place for mitigating the pain of migration for those who choose to migrate (no coercion). Just like 2to3 eases the transition from 2.x to 3.x, while forcing no-one to move over. I certainly don't believe the answer is to keep pkg_resources and setuptools APIs as some kind of fossilised, inviolable standard. I do believe we have to move away from custom installation code, which distutils by its nature forced people to produce, and which leads to problems for some people. Although I haven't published all my results, as it's still work in progress, I managed to extract all the metadata from thousands of packages hosted on PyPI from setup() to package.yaml, and then was able to generate a source archive from package.yaml which was semantically the same as the original. So I'm optimistic that by working on refining the metadata extraction mechanism, most of the packages on PyPI can be represented by an alternative metadata format, which allows the existence of a multiplicity of solutions to build, package and install Python software. > It is wonderful to have distlib. I support it. I'm playing with the > competing distlib2 implementation so that both APIs can be better, so > we can find out which parts provide functionality that does not just > have a different name in pkg_resources, and so that it can be possible > to replace the implementation without changing the API. If your goal > is to avoid "implementation defined behavior" it's a good idea to have > two. Let a thousand flowers bloom :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Oct 5 20:28:01 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 18:28:01 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Paul Moore gmail.com> writes: > - a reasonable proportion of distlib offers alternatives to the > pkg_resources code, and more could be added. The next thing for me to look at is "entry points", which should be relatively straightforward to implement. Regards, Vinay Sajip From p.f.moore at gmail.com Fri Oct 5 20:38:47 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 19:38:47 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On 5 October 2012 16:40, Daniel Holth wrote: >> Well, XML-RPC is of course preferable; the current code in distlib is just >> whatever I copied across from packaging, but the next step will be to look >> at the releases which are available from the different sources (XML-RPC, >> PyPI metadata URLs, dependency_links etc.) to see what sorts of things wouldn't >> be accessible if we restricted to say, just using XML-RPC. Since all the >> information in the static pages seems to be available via XML-RPC, what is the >> point of the simple interface, other than for occasional viewing by a human? > > IIRC the most practical limitation is that the XML-RPC interface > doesn't exist on the mirrors. That's a good point. Actually, writing a "local PyPI server" is much easier if all you have to implement is the simple static page interface. So I take back some of my objection - both XML-RPC and the static page interface make sense to support. Although it would be nice to have a better definition of precisely what comprises the "simple interface" than looking at the source code of the scraper. (Oh, and I still think the actual PyPI static pages include more links than are necessary, but that's a different issue, and one which would be alleviated by an option to ignore offsite links). Paul. From vinay_sajip at yahoo.co.uk Fri Oct 5 20:40:14 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 5 Oct 2012 18:40:14 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: Daniel Holth gmail.com> writes: > I'm basing this only on Vinay's numbers of > how-many-packages-can-generate-a-yaml. It's probably mostly packages > that import something he didn't have installed inside setup.py, but I > don't have a good way to find out exactly what is wrong with each one. > An awful lot of packages do work fine; I don't have an automated way > to detect that either. See this gist of the current crop of errors: https://gist.github.com/3841557 There are 1609 lines, after processing 17956 PyPI archives. Of the failures, 306 are because the source archive I generated from package.yaml didn't match the results of "python setup.py sdist". The remaining 1303 are failures to generate package.yaml. A depressing number are because the packagers failed to include files like 'README.rst' or 'CHANGES' in the distribution, but setup.py bails if they aren't there. Some of these will be false positives, i.e. bugs in my code. But many of the ones I've investigated hold up as a problem with the archive on PyPI. Regards, Vinay Sajip From dholth at gmail.com Fri Oct 5 21:22:25 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 15:22:25 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 2:40 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> I'm basing this only on Vinay's numbers of >> how-many-packages-can-generate-a-yaml. It's probably mostly packages >> that import something he didn't have installed inside setup.py, but I >> don't have a good way to find out exactly what is wrong with each one. >> An awful lot of packages do work fine; I don't have an automated way >> to detect that either. > > See this gist of the current crop of errors: > > https://gist.github.com/3841557 > > There are 1609 lines, after processing 17956 PyPI archives. Of the failures, > 306 are because the source archive I generated from package.yaml didn't match > the results of "python setup.py sdist". The remaining 1303 are failures to > generate package.yaml. A depressing number are because the packagers failed to > include files like 'README.rst' or 'CHANGES' in the distribution, but setup.py > bails if they aren't there. > > Some of these will be false positives, i.e. bugs in my code. But many of the > ones I've investigated hold up as a problem with the archive on PyPI. Neat. Packages that forgot to include README.rst in the sdist aren't that interesting. Tragic though. What would be really cool would be a web interface to all the setup.py on pypi. Detect whether the sdist definitely does not use conditional dependencies and ignore. If it might use conditional dependencies, interested Pythoneers read setup.py and type them in. Installer checks out of band metadata to install the build deps and make better decisions about the regular deps. From sklein at cpcug.org Fri Oct 5 21:28:38 2012 From: sklein at cpcug.org (Stanley A. Klein) Date: Fri, 5 Oct 2012 15:28:38 -0400 (EDT) Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: Message-ID: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> The issue I had raised was attention to the needs of Linux packaging and the Filesystem Hierarchy Standard, and apparently that is under consideration. The response I received (from "Daniel Holth" ) said in part "The FHS issues are on the list, which is why there are "resource categories" that can be installed wherever in new-packaging land." I assume that by an "installer" you mean the combination of package building (done by e.g., rpmbuild) and installation (done by e.g., rpm or yum). The mapping of files from the Pypi structure to the target directories is done at package-build time and not usually at install-time. What a function like bdist_rpm does is to create an rpm spec and turn the package building function over to rpmbuild. The file mapping is defined in the rpm spec and takes place during the rpmbuild processing. What I didn't notice in the distlib documentation were low level functions that would facilitate the allocation of Pypi files/directories to target files/directories to help in preparation of the rpm spec. Stan Klein On Fri, October 5, 2012 1:24 pm, Vinay Sajip wrote: > Message: 3 > Date: Fri, 5 Oct 2012 15:49:04 +0000 (UTC) > From: Vinay Sajip > To: Distutils-Sig at Python.Org > Subject: Re: [Distutils] distlib updated - comments sought > > Stanley A. Klein cpcug.org> writes: > >> I looked at the documentation and can't completely follow what you are >> doing, although I was told in another email that the issue I raised was >> being addressed in pkg_resources. > > The goal of distlib is to provide a library of low-level functions > relating > to packaging and distribution of Python software, and it does not contain > any > installers (though it will contain low-level code which could be used by > an > installer). The distlib code aims to serve as an implementation of the > various > packaging PEPs, such that if higher level tools use distlib, they should > be > able to both save writing code (and thereby gain time) as well as achieve > better interoperability with other packaging tools. > > The goal of distlib is not to be a complete packaging solution (which > packaging > tried to be for Python 3.3, but it proved to be too ambitious a goal). > > Which issue are you referring to, which you were told was addressed by > pkg_resources? From the section starting "Here is what I need", I couldn't > see how pkg_resources could fulfil those needs by itself. > > >> Here is what I need: > [snip] >> I can't completely see how I would do this using what is described in >> the >> distlib documentation. > > What you need is an installer, and distlib doesn't contain functionality > to do > installation. I won't rehash the various discussions around the needs of > Linux > distro packagers, except to say that the PEPs have taken into account the > needs > of distro packagers and the ability to map various parts of a distribution > to > various target directories in a distro-specific manner. > > Regards, > > Vinay Sajip > From p.f.moore at gmail.com Fri Oct 5 22:13:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 5 Oct 2012 21:13:11 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> References: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> Message-ID: On 5 October 2012 20:28, Stanley A. Klein wrote: > What I didn't notice in the distlib documentation were low level functions > that would facilitate the allocation of Pypi files/directories to target > files/directories to help in preparation of the rpm spec. While I'm not particularly familiar with Linux filesystems and FHS, I believe the key piece of the puzzle is in the stdlib, the sysconfig module. The "PyPI files/directories" as you call them (actually, the Python standard locations) are not specific directories, but rather are given by logical "paths" - platlib, scripts, data, and so on. Mapping these to OS-dependent locations can be handled by the OS installer code (RPM, I guess, in the case of Red Hat and similar). Properly written Python packages will only use those standard names. I hope that helps. Paul. From dholth at gmail.com Fri Oct 5 22:36:40 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 5 Oct 2012 16:36:40 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> References: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> Message-ID: On Fri, Oct 5, 2012 at 3:28 PM, Stanley A. Klein wrote: > The issue I had raised was attention to the needs of Linux packaging and > the Filesystem Hierarchy Standard, and apparently that is under > consideration. > > The response I received (from "Daniel Holth" ) said in > part "The FHS issues are on the list, which is why there are "resource > categories" that can be installed wherever in new-packaging land." > > I assume that by an "installer" you mean the combination of package > building (done by e.g., rpmbuild) and installation (done by e.g., rpm or > yum). > > The mapping of files from the Pypi structure to the target directories is > done at package-build time and not usually at install-time. What a > function like bdist_rpm does is to create an rpm spec and turn the package > building function over to rpmbuild. The file mapping is defined in the > rpm spec and takes place during the rpmbuild processing. > > What I didn't notice in the distlib documentation were low level functions > that would facilitate the allocation of Pypi files/directories to target > files/directories to help in preparation of the rpm spec. I don't think that is in distlib's problem domain at the moment; the information about which file should go where is not generally present in setup.py at all. It was part of the distutils2/packaging effort to define a setup.cfg [files] section. Similar to http://docs.python.org/distutils/setupscript.html#installing-additional-files but with better definitions for the category names, and a config file to show where they should go. Bento uses the autoconf names http://cournape.github.com/Bento/html/tutorial.html#adding-data-files, http://cournape.github.com/Bento/html/reference/bento_format.html#available-path-variables, e.g. prefix: install architecture-independent files eprefix: install architecture-dependent files bindir: user executables sbindir: system admin executables ... The next build system that's made, or Bento right now, will provide for finding those files again at runtime even if they have been flung to the four corners of the disk. The wheel spec should probably be updated to understand something about packagename-1.0.data/bindir etc. When you are build an RPM, install-time from the Python perspective would indeed be build-time from the RPM packager's perspective. Daniel From a.cavallo at cavallinux.eu Sat Oct 6 09:44:53 2012 From: a.cavallo at cavallinux.eu (Antonio Cavallo) Date: Sat, 6 Oct 2012 08:44:53 +0100 Subject: [Distutils] distlib updated - comments sought In-Reply-To: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> References: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> Message-ID: On 5 Oct 2012, at 20:28, Stanley A. Klein wrote: > I assume that by an "installer" you mean the combination of package > building (done by e.g., rpmbuild) and installation (done by e.g., rpm or > yum). Hi, I think you've hit the jackpot here? Here few lesson learnt using rpm and python at fairly large scale (http://download.opensuse.org/repositories/home:/cavallo71:/opt-python-interpreters) project of mine I wish could be of some help in thinking of a "packaging system". (I'm totally happy with the old-pain setup.py and I see no need to switch to anything else). * Installation != Configuration * running scripts at "install" time transforms the installer into a development process (I'm finger pointing the retarded dpkg debian way). Good rpm packages don't have *any* script at all. * Don't cross roles border * Files don't get created into the void: the act of "installing" shouldn't be no more than transferring file to a filesystem plus adding metadata information in a database eventually (the rpm database, the windows registry etc.). Adjusting paths/files during the install (msiexec /i *.msi, rpm -i *.rpm etc.) is not the right way. * I'm not expecting tar to download packages from internet, so rpm * Having dependency resolution (and trying to install packages within a setup.py of any sort) is outright retarded although seems to fit somebody agenda. There're already plenty of tools in the os to do that e.g.. the dependency resolution between rpms is handled at yum level with a tiny help from the rpm. I don't want that placed in the wrong place. BTW rpmbuild is a tool part of rpm, technically users (as sysadmin) shouldn't needed for managing packages. From doko at ubuntu.com Sat Oct 6 17:28:28 2012 From: doko at ubuntu.com (Matthias Klose) Date: Sat, 06 Oct 2012 17:28:28 +0200 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: <50704E1C.9000606@ubuntu.com> On 05.10.2012 19:24, Paul Moore wrote: > On 5 October 2012 17:04, Daniel Holth wrote: > >> ~1300 of the ~20000 packages on pypi have trouble using setup.py as >> their build system / metadata source format. > > That's interesting information. Do you know in what way they have > trouble with setup.py? Do they not use it at all, do they need > features it doesn't provide, or what? > >> For the ~1300 broken packages, distutils is awful because it is not >> really extensible, though setuptools tried. > > Yeah, that's the common complaint. Plus, "it's too extensible" :-) > (From people trying to change it who have to deal with all the fancy > hacks people have done). > >> People have to install setuptools against their will because there is >> only one implementation of the pkg_resources API and 75% of the >> packages on pypi require setuptools. > > I wish we could separate pkg_resources and setuptools. I'd love to > know which packages needed each (but I suspect that's not a question > that can be answered without looking at the actual code). Ignoring the > egg support aspects, pkg_resources is something that could be replaced > - a reasonable proportion of distlib offers alternatives to the > pkg_resources code, and more could be added. On the other hand, > setuptools per se is almost entirely a build time facility, so > shouldn't be needed at runtime (and so using it for build should be > relatively unimportant). Debian and Ubuntu have a (deb) binary package python-pkg-resources. 817 packages do use python-setuptools for the build, and 340 binary packages do depend on python-pkg-resources. so it's not just a few, but nearly 50%. see apt-cache rdepends python-pkg-resources for the full list. Matthias PS: are there really ~20000 packages on pypi, or do you count old versions too? From vinay_sajip at yahoo.co.uk Sat Oct 6 23:39:23 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 6 Oct 2012 21:39:23 +0000 (UTC) Subject: [Distutils] distlib updated - comments sought References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> <50704E1C.9000606@ubuntu.com> Message-ID: Matthias Klose ubuntu.com> writes: > > PS: are there really ~20000 packages on pypi, or do you count old versions too? > Not counting old versions, my current list of packages where an archive is hosted on PyPI numbers a little under 18,000. While not bang up to date, it's unlikely to be too far out. There are additional packages which are listed on PyPI but not hosted there. I added a file, archives.txt, to the Gist at https://gist.github.com/3841557 This file has all the archives I processed - you can't see the text in the Gist (it's too big, 17956 lines), but the raw file is available via a link. Regards, Vinay Sajip From dholth at gmail.com Mon Oct 8 18:40:48 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 8 Oct 2012 12:40:48 -0400 Subject: [Distutils] [ANN] wheel@librelist.com mailing list Message-ID: I've started a list wheel at librelist.com to discuss the implementation of the 'wheel' binary package format, instead of discussing it all on this list. Mail the list to join. We are hoping to have support merged into pip soon. Daniel H From dholth at gmail.com Mon Oct 8 19:42:22 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 8 Oct 2012 13:42:22 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <51829.71.163.193.12.1349465318.squirrel@www.cpcug.org> Message-ID: On Sat, Oct 6, 2012 at 3:44 AM, Antonio Cavallo wrote: > Here few lesson learnt using rpm and python at fairly large scale (http://download.opensuse.org/repositories/home:/cavallo71:/opt-python-interpreters) project of mine I wish could be of some help in thinking of a "packaging system". > (I'm totally happy with the old-pain setup.py and I see no need to switch to anything else). I agree that setup.py should be replaced and that Python packaging should be better in various ways, but I think the process of replacing the build system should take 5-10 years. I do not want to fork the 27 working dependencies of my simplest project by rewriting their setup.py and replacing any "import pkg_resources" with a different API. Instead, I will use binary packaging to forget about the build system at install time and to install much more quickly, and if setup.py is a problem, at least it's a problem less often (because 'install' is a much more common operation than 'build'). If it becomes necessary, I (or someone) will re-implement the most used parts of the pkg_resources API with a different back end. The strategy has nothing to do with liking or disliking the existing popular APIs. It is only because it takes less time than debugging and maintaining forks of my dependencies due to a subtly incompatible new build system. (If the new build system was not incompatible, it would not be able to improve on setuptools and distutils.) Eventually everyone will realize that we should be building with Bento, and we will get better APIs that do what pkg_resources does without confusing people with a very long single source file and the new systems will become preferred. In the meantime there is going to be more than one way to do it for a long time. From vinay_sajip at yahoo.co.uk Mon Oct 8 20:05:46 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 8 Oct 2012 18:05:46 +0000 (UTC) Subject: [Distutils] distlib updated to include "entry point" functionality Message-ID: I've now updated distlib to include equivalent functionality for pkg_resources' entry points. There is no ability to install packages on-demand via "require()" functionality, but distributions are able to register things they implement, and the capability to query what's available is also present. The relevant information is stored in a dist-info file REGISTRY which has the same format as entry_points.txt files. There is no code at present to write these files; that will come later when focusing on what installer software will need from distlib. The distlib.depgraph code has now been merged into distlib.database, as indicated in an earlier post. I've also added a section on migrating from older APIs. This currently covers pkg_resources' resource and entry point APIs. The resources API now covers extraction of files from .zip files and caching of the extracted files in the file system. I noticed that pkg_resources has a function cleanup_resources() to clear the cache, but this is a no-op. Can someone tell me why this is? Regards, Vinay Sajip From pje at telecommunity.com Mon Oct 8 20:25:04 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 8 Oct 2012 14:25:04 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Mon, Oct 8, 2012 at 2:05 PM, Vinay Sajip wrote: > I've now updated distlib to include equivalent functionality for pkg_resources' > entry points. There is no ability to install packages on-demand via "require()" > functionality, but distributions are able to register things they implement, > and the capability to query what's available is also present. The relevant > information is stored in a dist-info file REGISTRY which has the same format as > entry_points.txt files. Given that it's the same format, is there any reason to change the filename? Changing the filename just means more code in Daniel's wheel support code, and REGISTRY is an awfully vague name. Registry of what? ;-) > I noticed that pkg_resources has a function cleanup_resources() to clear the > cache, but this is a no-op. Can someone tell me why this is? When Bob Ippolito and I first designed that system, we thought there'd be a need to use temporary directories for extraction. Apparently, there never was. ;-) From jaraco at jaraco.com Mon Oct 8 20:05:26 2012 From: jaraco at jaraco.com (Jason R. Coombs) Date: Mon, 8 Oct 2012 18:05:26 +0000 Subject: [Distutils] distribute buildbot In-Reply-To: <503DF240.2010201@gmail.com> References: <50374470.40105@gmail.com> <503DF240.2010201@gmail.com> Message-ID: <7E79234E600438479EC119BD241B48D623CB0FB2@BL2PRD0611MB422.namprd06.prod.outlook.com> Also, I've started mirroring distribute at github and running the tests using travis-ci (https://travis-ci.org/#!/jaraco/distribute). I haven't looked into why the tests are currently failing. They were passing at one point, though they seem to fail intermittently. > -----Original Message----- > From: distutils-sig-bounces+jaraco=jaraco.com at python.org [mailto:distutils- > sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Adam GROSZER > Sent: Wednesday, 29 August, 2012 06:43 > To: Distutils-SIG > Subject: Re: [Distutils] distribute buildbot > > To answer myself, winbot.zooe.org builds now distribute > > On 08/24/2012 11:08 AM, Adam GROSZER wrote: > > Hello, > > > > Are there any buildbots for distribute? > > > > > -- > Best regards, > Adam GROSZER > -- > Quote of the day: > Rudin's Law: In a crisis that forces a choice to be made among alternative > courses of action, most people will choose the worst one possible. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6662 bytes Desc: not available URL: From jtayon at gmail.com Mon Oct 8 20:46:46 2012 From: jtayon at gmail.com (Julien Tayon) Date: Mon, 8 Oct 2012 20:46:46 +0200 Subject: [Distutils] distribute buildbot In-Reply-To: <7E79234E600438479EC119BD241B48D623CB0FB2@BL2PRD0611MB422.namprd06.prod.outlook.com> References: <50374470.40105@gmail.com> <503DF240.2010201@gmail.com> <7E79234E600438479EC119BD241B48D623CB0FB2@BL2PRD0611MB422.namprd06.prod.outlook.com> Message-ID: Because when travis build farm is overloaded, and hence it cannot launch a testing environment, it says your tests failed. Le 8 oct. 2012 ? 20:05, "Jason R. Coombs" a ?crit : > Also, I've started mirroring distribute at github and running the > tests > using travis-ci (https://travis-ci.org/#!/jaraco/distribute). > > I haven't looked into why the tests are currently failing. They were > passing > at one point, though they seem to fail intermittently. > >> -----Original Message----- >> From: distutils-sig-bounces+jaraco=jaraco.com at python.org > [mailto:distutils- >> sig-bounces+jaraco=jaraco.com at python.org] On Behalf Of Adam GROSZER >> Sent: Wednesday, 29 August, 2012 06:43 >> To: Distutils-SIG >> Subject: Re: [Distutils] distribute buildbot >> >> To answer myself, winbot.zooe.org builds now distribute >> >> On 08/24/2012 11:08 AM, Adam GROSZER wrote: >>> Hello, >>> >>> Are there any buildbots for distribute? >>> >> >> >> -- >> Best regards, >> Adam GROSZER >> -- >> Quote of the day: >> Rudin's Law: In a crisis that forces a choice to be made among >> alternative >> courses of action, most people will choose the worst one possible. >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> http://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Mon Oct 8 21:34:38 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 8 Oct 2012 19:34:38 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: PJ Eby telecommunity.com> writes: > > Given that it's the same format, is there any reason to change the > filename? Changing the filename just means more code in Daniel's > wheel support code, I'm not too hung up about the filename, and only chose REGISTRY because it is more aligned with the other filenames which appear in .dist-info directories. Since distlib is still in its very early days, I expect the filename can be changed without too much of an impact; the idea is to hide that sort of implementation detail behind an API. > and REGISTRY is an awfully vague name. Registry > of what? Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":" > When Bob Ippolito and I first designed that system, we thought there'd > be a need to use temporary directories for extraction. Apparently, > there never was. Thanks for the info, and the feedback. Regards, Vinay Sajip From regebro at gmail.com Mon Oct 8 21:59:47 2012 From: regebro at gmail.com (Lennart Regebro) Date: Mon, 8 Oct 2012 21:59:47 +0200 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip wrote: > Registry of what ever you like - pointers to data, perhaps. Any use of the > registry, just like entry points, is a matter of convention between consenting > / cooperating developers and their applications/libraries, or am I missing > something? "Entry points" are not always entry points, are they? IIUC, you're > allowed to just define a module or package without the attributes appearing > after the ":" Right, but isn't that then an entry point? I agree REGISTRY is not a very good name in any case. From vinay_sajip at yahoo.co.uk Mon Oct 8 22:56:48 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 8 Oct 2012 20:56:48 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: Lennart Regebro gmail.com> writes: >> IIUC, you're allowed to just define a module or package without the >> attributes appearing after the ":" > > Right, but isn't that then an entry point? > How is it an entry point? Perhaps I'm misunderstanding what an entry point is, but to my mind it's the start of some code. Of course a module runs code when imported, but the module may already have been imported when the entry point is loaded - in which case, nothing would get run at load time, IIUC. Even if it wasn't imported before, I'm not sure whether you'd want to import a module just for side-effects caused by the import. > I agree REGISTRY is not a very good name in any case. Like I said, I only chose it because it's similar to other filenames in .dist-info, and I'll entertain alternative bikeshed colours ;-) I've added read_registry and write_registry methods to Distribution which take an optional filename; this allows reading from alternative filenames, which is handy for testing, at least. Regards, Vinay Sajip From dholth at gmail.com Mon Oct 8 23:03:15 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 8 Oct 2012 17:03:15 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Mon, Oct 8, 2012 at 3:59 PM, Lennart Regebro wrote: > On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip wrote: >> Registry of what ever you like - pointers to data, perhaps. Any use of the >> registry, just like entry points, is a matter of convention between consenting >> / cooperating developers and their applications/libraries, or am I missing >> something? "Entry points" are not always entry points, are they? IIUC, you're >> allowed to just define a module or package without the attributes appearing >> after the ":" > > Right, but isn't that then an entry point? > > I agree REGISTRY is not a very good name in any case. I would prefer to keep the filename the same as long as the format is the same. (Not a suggestion to change the format.) bdist_wheel is easy to edit but pkg_resources.py is not because it has a long release cycle and if you make a mistake you will break almost every Python installation. Right now a new class attribute Distribution.PKG_INFO keeps track of whether the PEP-defined Metadata 1.0-1.3 is called PKG-INFO or METADATA. To support REGISTRY in pkg_resources you might do something similar, changing line ~2319 self._get_metadata('entry_points.txt'), self to self._get_metadata(self.ENTRY_POINTS_FILENAME), self or for name in (possible, entry, points, filenames): ... defining ENTRY_POINTS_FILENAME for both Distribution and DistInfoDistribution, and you would have to wait for a new distribute release. In other words, it is a hassle and the feature is the same. An entry_point is always an entry_point. It should no sooner be confused with an English "point of entry" than keywords like "print" should be confused with pressing books. Daniel From vinay_sajip at yahoo.co.uk Mon Oct 8 23:14:51 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 8 Oct 2012 21:14:51 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: Daniel Holth gmail.com> writes: > I would prefer to keep the filename the same as long as the format is > the same. See my comments about read_registry and write_registry methods in Distribution (in response to Lennart's post). > In other words, it is a hassle and the feature is the same. An > entry_point is always an entry_point. It should no sooner be confused > with an English "point of entry" than keywords like "print" should be > confused with pressing books. What do *you* mean by entry point? My understanding is that it represents the start of some executable code. Wikipedia: "In computer programming, an entry point is a memory address, corresponding to a point in the code of a computer program which is intended as the destination of a long jump, be it internal or external." So, in the Python sphere, I would interpret an entry point to be a callable. While that might be the most common usage of the feature, it's not the only one. For example, you could use name = value entries where value is not a callable, but for example a name of a resource to use for something, or a configuration option, or the name of a package or a module (which is not an entry point by my understanding of entry points being callables). Regards, Vinay Sajip From dholth at gmail.com Tue Oct 9 20:24:09 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 9 Oct 2012 14:24:09 -0400 Subject: [Distutils] semver (again) Message-ID: I like semver. My favorite part is point 12: Precedence MUST be calculated by separating the version into major, minor, patch, pre-release, and build identifiers in that order. Major, minor, and patch versions are always compared numerically. Pre-release and build version precedence MUST be determined by comparing each dot separated identifier as follows: identifiers consisting of only digits are compared numerically and identifiers with letters or dashes are compared lexically in ASCII sort order. Numeric identifiers always have lower precedence than non-numeric identifiers. Example: 1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0-rc.1+build.1 < 1.0.0 < 1.0.0+0.3.7 < 1.3.7+build < 1.3.7+build.2.b8f12d7 < 1.3.7+build.11.e0f985a. (No non-lexicographic comparisons). Unlike PEP-386 it would also allow DVCS revision ids to be included in the version string. Taking a look at pypi, ~1k of the latest releases do not parse as semver (if you extend it by making 1.0 equivalent to 1.0.0, as all semver versions have at least (major, minor, patch)). This is similar to the number of latest package versions that don't parse as PEP-386. I also found this blog post. http://blog.ziade.org/2009/07/03/dropping-pep-386-versions-comparison/ How much would it cost to endorse semver? Daniel H From fdrake at acm.org Tue Oct 9 20:43:40 2012 From: fdrake at acm.org (Fred Drake) Date: Tue, 9 Oct 2012 14:43:40 -0400 Subject: [Distutils] semver (again) In-Reply-To: References: Message-ID: On Tue, Oct 9, 2012 at 2:24 PM, Daniel Holth wrote: > How much would it cost to endorse semver? I endorse it for free. :-) Seriously, if a version doesn't conform, I probably don't want to be using that piece of software, because it's versioning model is overly complicated. Integers are cheaper than these discussions; just increment the right one. Win: No more "dev" builds. Of course, I'm not known for being sympathetic regarding software. -Fred -- Fred L. Drake, Jr. "A person who won't read has no advantage over one who can't read." --Samuel Langhorne Clemens From pje at telecommunity.com Tue Oct 9 22:29:41 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 9 Oct 2012 16:29:41 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Mon, Oct 8, 2012 at 5:14 PM, Vinay Sajip wrote: > What do *you* mean by entry point? My understanding is that it represents the > start of some executable code. Wikipedia: > > "In computer programming, an entry point is a memory address, corresponding to a > point in the code of a computer program which is intended as the destination of a > long jump, be it internal or external." > > So, in the Python sphere, I would interpret an entry point to be a callable. > While that might be the most common usage of the feature, it's not the only one. > For example, you could use name = value entries where value is not a callable, > but for example a name of a resource to use for something, or a configuration > option, or the name of a package or a module (which is not an entry point by my > understanding of entry points being callables). In pkg_resources, entry points are references to any importable object. They don't have to be callables. Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard. ;-) In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.) If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import. I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times. That's why I baked those features into entry points from the beginning - I was battling what was at the time an excruciating slow startup for the Chandler application platform, and I borrowed the architectural pattern of delayed loading of plugins from Eclipse. (I also remember that Ian Bicking was talking about some problem, maybe on the Web-SIG, and I remember that the problem he was talking about could be solved by something like entry points, at which point I then had a solution for both Ian's problem and Chandler's problem.) From vinay_sajip at yahoo.co.uk Tue Oct 9 23:45:33 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 9 Oct 2012 21:45:33 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: PJ Eby telecommunity.com> writes: > > In pkg_resources, entry points are references to any importable > object. They don't have to be callables. > > Note, however, that if you load an entry point which is a module, then > of course the module code will be executed as a side-effect, so > technically, a module still meets the "start of some executable code" > standard. Yes, I acknowledged this in my response to Lennart's post. > In any case, the name was chosen for the primary use case of > designating executable entry points in Python code, often callables. > However, given the nature of Python, there was absolutely no reason to > restrict them to referring *only* to callables. (In fact, the > feature's name and implementation was inspired by Eclipse's "extension > points" architecture, where the advertised objects usually implement > interfaces with more than one method.) Right, they're just objects of one sort or another. > > If the name *must* be changed, I'd suggest using "exports", for > symmetry with "imports". Entry points are always the name of an > importable object, and they are basically being advertised so that > other code knows what is available to be imported, and the loading API > allows one to handle other details such as updating sys.path as > necessary, and doing the actual import. As you say, entry points don't have to be actual entry points, and I didn't want to slavishly follow pkg_resources nomenclature; though it reflects the most common use case, it's not the only one. Your suggestion of "exports" seems very reasonable as a replacement for "registry", which I agree is too generic. > I understand that right now you're not supporting those other features > in distlib, but I feel I should mention as a reminder that many of > these pkg_resources features were created to support application > platforms with plugins, such as Trac, Zope, Chandler, etc., and to > prevent the need for wheel-reinvention between different Python plugin > systems. For this type of platform, it's often an important core > feature to be able to dynamically add plugins and their requirements > to sys.path on an as-needed basis when the entry points are actually > *used*, because early loading or additions to sys.path can > dramatically affect application startup times. I understand, and appreciate your pointing these things out. As I see it, the specific way that some of those features were implemented is what some people object to. In distlib, importing is still lazy, but sys.path is not updated before attempting imports. Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib, whereas distlib hopes to provide (some of) the useful features provided in those packages in a (hopefully) non-contentious way, with perhaps part of it going into the stdlib at some future date if there's a consensus about that. ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path. Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned? Regards, Vinay Sajip From pje at telecommunity.com Wed Oct 10 00:56:02 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 9 Oct 2012 18:56:02 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip wrote: > Clearly, distlib can't do things exactly > the way they're done in pkg_resources and setuptools, since those are not > considered suitable to go into the stdlib, The two aren't 100% connected, even assuming the antecedent were true. Originally, Guido Pronounced that they *were* to be added to the stdlib (for Python 2.5), and I withdrew them due largely to an objection about the lack of third-party expertise in the underlying formats -- an expertise that is now considerably more widespread, especially thanks to you and Daniel. ;-) More to the point, though, if it's not possible for someone to create a wrapper to reimplement these portions of the pkg_resources API, then it's going to be harder to get people to port their code. The challenge isn't necessarily that there are dozens of platforms that rely on this feature explicitly; the challenge is that code which uses pkg_resources APIs for these things is *intentionally ignorant* of whether the feature is being used. The entire point of the API is to free the caller from worrying about the callee's needs, and the callee's dependencies can be transparently met by distributions that are not directly on sys.path, at least in the .egg or .wheel model of things. (If .wheel files are importable, anyway.) > ISTM that the "exports" feature in distlib allows applications to cooperate in > the same way as pkg_resources' entry points do, except for the fact that > sys.path is not manipulated at all, meaning that all such cooperating > applications are assumed to be installed on the path. Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the extras required by an entry point? (i.e., is it possible for someone to make a common wrapper that re-adds sys.path management?) > Actual importing is > delayed until needed; scanning of distribution metadata for exports also > happens on demand. Do you feel that this is insufficient to meet the > performance concerns you mentioned? Now that 3.3 includes a faster import mechanism, probably so. Under older Pythons, adding a large number of directories to sys.path could badly affect startup performance, if plugins were installed as directories instead of as zipfiles. (As a general note, in case you aren't aware: the entire .egg format was invented originally as a distribution+use format for application plugins; Bob Ippolito suggested that in the general case, any Python library could also be considered a "plugin" to Python itself, and then things kind of took off from there...) From pje at telecommunity.com Wed Oct 10 01:10:42 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 9 Oct 2012 19:10:42 -0400 Subject: [Distutils] distlib updated - comments sought In-Reply-To: References: <1349429254.56868.YahooMailNeo@web171404.mail.ir2.yahoo.com> Message-ID: On Fri, Oct 5, 2012 at 2:38 PM, Paul Moore wrote: > That's a good point. Actually, writing a "local PyPI server" is much > easier if all you have to implement is the simple static page > interface. So I take back some of my objection - both XML-RPC and the > static page interface make sense to support. Although it would be nice > to have a better definition of precisely what comprises the "simple > interface" than looking at the source code of the scraper. *cough* http://peak.telecommunity.com/DevCenter/EasyInstall#package-index-api http://peak.telecommunity.com/DevCenter/setuptools#making-your-package-available-for-easyinstall Admittedly, not all of the specific heuristics for determining what kind of distribution a file is are documented; the rules for identifying distutils sdist and bdist_dumb files are unfortunately somewhat arcane, because distutils' filename conventions are inherently ambiguous for parsing. Setuptools' rules for unambiguous filename generation and parsing, however, are documented here: http://peak.telecommunity.com/DevCenter/EggFormats#filename-embedded-metadata http://peak.telecommunity.com/DevCenter/PkgResources#parsing-utilities > (Oh, and I > still think the actual PyPI static pages include more links than are > necessary, but that's a different issue, and one which would be > alleviated by an option to ignore offsite links). See also: http://peak.telecommunity.com/DevCenter/EasyInstall#restricting-downloads-with-allow-hosts ;-) From vinay_sajip at yahoo.co.uk Wed Oct 10 01:28:05 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 9 Oct 2012 23:28:05 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: PJ Eby telecommunity.com> writes: > More to the point, though, if it's not possible for someone to create > a wrapper to reimplement these portions of the pkg_resources API, then > it's going to be harder to get people to port their code. I will give some thought to migration concerns. I'm not sure exactly what the best approach is, or if identical semantics are achievable in all areas. > of whether the feature is being used. The entire point of the API is > to free the caller from worrying about the callee's needs, and the > callee's dependencies can be transparently met by distributions that > are not directly on sys.path, at least in the .egg or .wheel model of > things. (If .wheel files are importable, anyway.) I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer. > Do you mean that it is not possible to identify the entry points of an > arbitrary distribution? What about obtaining information about the At the moment, not for a completely arbitrary distribution, as we don't have the metadata in non-installed distributions. Of course, an arbitrary distribution could be unpacked to a temporary location and have the equivalent of egg-info run to get the information; but for this to work ideally, such information would need to be present in the archive metadata and also need to be in the index. There are a number of ways this could happen - it's not clear to me what the best way is, but I probably haven't thought about it enough. > extras required by an entry point? (i.e., is it possible for someone > to make a common wrapper that re-adds sys.path management?) I think it's too early to say. > > Actual importing is > > delayed until needed; scanning of distribution metadata for exports also > > happens on demand. Do you feel that this is insufficient to meet the > > performance concerns you mentioned? > > Now that 3.3 includes a faster import mechanism, probably so. Under > older Pythons, adding a large number of directories to sys.path could > badly affect startup performance, if plugins were installed as > directories instead of as zipfiles. I assume from the context above that that you actually mean "no" as the answer to the question "is it insufficient?", whereas I would take "probably so" to mean "yes, it's insufficient". Regards, Vinay Sajip From dholth at gmail.com Wed Oct 10 01:53:49 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 9 Oct 2012 19:53:49 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: > >> >> In pkg_resources, entry points are references to any importable >> object. They don't have to be callables. >> >> Note, however, that if you load an entry point which is a module, then >> of course the module code will be executed as a side-effect, so >> technically, a module still meets the "start of some executable code" >> standard. > > Yes, I acknowledged this in my response to Lennart's post. > >> In any case, the name was chosen for the primary use case of >> designating executable entry points in Python code, often callables. >> However, given the nature of Python, there was absolutely no reason to >> restrict them to referring *only* to callables. (In fact, the >> feature's name and implementation was inspired by Eclipse's "extension >> points" architecture, where the advertised objects usually implement >> interfaces with more than one method.) > > Right, they're just objects of one sort or another. > >> >> If the name *must* be changed, I'd suggest using "exports", for >> symmetry with "imports". Entry points are always the name of an >> importable object, and they are basically being advertised so that >> other code knows what is available to be imported, and the loading API >> allows one to handle other details such as updating sys.path as >> necessary, and doing the actual import. > > As you say, entry points don't have to be actual entry points, and I didn't > want to slavishly follow pkg_resources nomenclature; though it reflects the > most common use case, it's not the only one. Your suggestion of "exports" > seems very reasonable as a replacement for "registry", which I agree is too > generic. > >> I understand that right now you're not supporting those other features >> in distlib, but I feel I should mention as a reminder that many of >> these pkg_resources features were created to support application >> platforms with plugins, such as Trac, Zope, Chandler, etc., and to >> prevent the need for wheel-reinvention between different Python plugin >> systems. For this type of platform, it's often an important core >> feature to be able to dynamically add plugins and their requirements >> to sys.path on an as-needed basis when the entry points are actually >> *used*, because early loading or additions to sys.path can >> dramatically affect application startup times. > > I understand, and appreciate your pointing these things out. As I see it, > the specific way that some of those features were implemented is what some > people object to. In distlib, importing is still lazy, but sys.path is not > updated before attempting imports. Clearly, distlib can't do things exactly > the way they're done in pkg_resources and setuptools, since those are not > considered suitable to go into the stdlib, whereas distlib hopes to provide > (some of) the useful features provided in those packages in a (hopefully) > non-contentious way, with perhaps part of it going into the stdlib at some > future date if there's a consensus about that. > > ISTM that the "exports" feature in distlib allows applications to cooperate in > the same way as pkg_resources' entry points do, except for the fact that > sys.path is not manipulated at all, meaning that all such cooperating > applications are assumed to be installed on the path. Actual importing is > delayed until needed; scanning of distribution metadata for exports also > happens on demand. Do you feel that this is insufficient to meet the > performance concerns you mentioned? As far as my performance concerns are concerned, I just want to be able to avoid iterating through all the installed distributions twice for code that is using both APIs, which besides the porting hassle is why I might implement one API on top of the other implementation. I would also like to be able to store my installed package database in sqlite3 by implementing an appropriate distlib/pkg_resources backend and defining a standard post-(un)install "index this new package" hook but I doubt I will have the (unfunded) time. I'm sure sys.path manipulation could be implemented on top of a basic Distributions-from-path API to search for the entry points, without having to be integrated with it. FYI Ruby's gem seems to do this with the equivalent of a Python import hook (our sys.meta_path) and they do index the available gems in an actual not-just-a-bunch-of-files database. There is an interesting but long discussion at http://python.6.n6.nabble.com/Re-Python-Dev-PEP-365-Adding-the-pkg-resources-module-td1995421.html GvR starts off by wishing it wasn't 1700 lines long. It seems "easy to review" was a more important factor than the specific name of any particular feature, except for "easy_install" which was seen as too much of a setuptools brand name. Vinay wrote: I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer. > > Do you mean that it is not possible to identify the entry points of an > > arbitrary distribution? What about obtaining information about the > At the moment, not for a completely arbitrary distribution, as we don't > have the metadata in non-installed distributions. Of course, an arbitrary > distribution could be unpacked to a temporary location and have the equivalent > of egg-info run to get the information; but for this to work ideally, such > information would need to be present in the archive metadata and also need to > be in the index. There are a number of ways this could happen - it's not > clear to me what the best way is, but I probably haven't thought about it > enough. At this point of the runtime, the "arbitrary non-installed distribution" is just a dist that has been installed outside of sys.path. So the static metadata is there. Wheels are designed to be valid sys.path entries in the simple case. We don't encourage you to do so mostly because we don't want to include .pyc files for all the Pythons you might use, but it is handy for bootstrapping. python wheel-0.12.0-py2.py3-none-any.whl/wheel install wheel-0.12.0-py2.py3-none-any.whl installs wheel's own wheel with itself, even when pkg_resources.py is not installed. (You could do the same thing with PYTHONPATH=... python -m wheel install). You could imagine including with Python a similar bootstrap installer as a runnable .zip file (invent an extension .pya / .pyp?). Since the functionality is hidden in the .zip file, you would only have to commit to and argue about the command-line API instead of the much larger package management API. The bootstrap installer would install the real installer or build systems such as distutils, setuptools, Bento, distlib, providing that those projects were willing to publish themselves to pypi as pre-built wheel archives. If you want to play around with renaming the egg-info files, check out Bento's egg_utils.py line 106: https://github.com/cournape/Bento/blob/master/bento/commands/egg_utils.py#L106 which controls what each metadata file is named. file_table = { "pkg_info": "PKG-INFO", "sources": "SOURCES.txt", "install_requires": "requires.txt", "top_levels": "top_level.txt", "not_zip_safe": "not-zip-safe", "dependency_links": "dependency_links.txt", "entry_points": "entry_points.txt", "ipkg_info": "ipkg.info", } From pje at telecommunity.com Thu Oct 11 00:06:01 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 10 Oct 2012 18:06:01 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Tue, Oct 9, 2012 at 7:28 PM, Vinay Sajip wrote: > I don't believe that wheels are meant to be directly importable, but I could > be wrong about that. I am not sure why callee dependencies need to be > transparently met by distributions not on sys.path; Broadly speaking, and > leaving eggs out of the picture, I would expect a distribution to declare its > dependencies, and for these to be installed in the appropriate site-packages > by the installer. Except that installation doesn't always equal placement on a sys.path. Application platforms can and do distinguish between a plugin that is "installed" and one that is "activated" or "enabled". Again, the entire egg infrastructure was originally designed to support pluggable applications, which is part of why it's so annoying to Linux packagers at times. (In roughly the same way as Firefox is annoying to them for having .xpi's.) I'm not saying distlib must support all these plugin usecases *itself*, but if it solves its chosen usecases in a way that can't be *adapted by others* to handle the app platform use cases, then there's not going to be an appealing alternative to sticking with pkg_resources or its derivatives. > I assume from the context above that that you actually mean "no" as the > answer to the question "is it insufficient?", whereas I would take "probably > so" to mean "yes, it's insufficient". Oops - I misread that as "Do you feel that this is sufficient?". Sorry. From pje at telecommunity.com Thu Oct 11 00:15:21 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 10 Oct 2012 18:15:21 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Tue, Oct 9, 2012 at 7:53 PM, Daniel Holth wrote: > would also like to be able to store my installed package database in > sqlite3 by implementing an appropriate distlib/pkg_resources backend > and defining a standard post-(un)install "index this new package" hook > but I doubt I will have the (unfunded) time. How would a database even work in that scenario (given multiple independent directories on sys.path), and why would you expect it to perform better than a modern filesystem? > There is an interesting but long discussion at > http://python.6.n6.nabble.com/Re-Python-Dev-PEP-365-Adding-the-pkg-resources-module-td1995421.html > > GvR starts off by wishing it wasn't 1700 lines long. It seems "easy to > review" was a more important factor than the specific name of any > particular feature, except for "easy_install" which was seen as too > much of a setuptools brand name. He also said egg support in the stdlib was controversial, in part because there were so many individual things that would be being blessed. However, there are now PEPs covering a lot of those things, like versioning standards, metadata, etc. In any case, I doubt that distlib would be a problem just because it can be *extended* to support eggs. In fact, he was fine with PEP 365 supporting *installing* setuptools, just not *including* setuptools. > Wheels are designed to be valid sys.path entries in the simple case. > We don't encourage you to do so mostly because we don't want to > include .pyc files for all the Pythons you might use, Wait... isn't wheel a binary format? Oh, I see what you mean, wheels are a *fat* binary format, so they can include multiple Python versions and multiple platforms. However, for the application platform plugins use case, wheels can potentially be quite awesome, because you can build one fat wheel for all your supported platforms. "We don't want to inlcude .pyc files for all the Pythons you might use" doesn't apply to the "we" who already build and release different platform-specific eggs to cover multiple platforms and Python versions. ;-) From p.f.moore at gmail.com Thu Oct 11 00:30:20 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 10 Oct 2012 23:30:20 +0100 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On 10 October 2012 23:15, PJ Eby wrote: > However, for the application platform plugins use case, wheels can > potentially be quite awesome, because you can build one fat wheel for > all your supported platforms. "We don't want to inlcude .pyc files > for all the Pythons you might use" doesn't apply to the "we" who > already build and release different platform-specific eggs to cover > multiple platforms and Python versions. ;-) I really don't understand the whole idea of including .pyc files in binary distributions - all they do is *reduce* the flexibility of the distribution by tying it to a single Python version. Far better to just ship the .py files and let the system compile them on demand (or if you absolutely insist, compile them at install time). Note - wheels are a binary *distribution* format, and as such aren't intended to be placed on sys.path directly. The fact that they can be is not important, that's just a side effect of using a zipfile with a simple internal layout. I concede that things are different for eggs which were intended to be an importable format (AIUI, more so than a distribution format, even). Of course, "fat" wheels containing multiple versions of *compiled* code (maybe for different architectures, or API versions) are conceivable - although I don't think the current internal layout of the wheel format supports that. Paul. From vinay_sajip at yahoo.co.uk Thu Oct 11 00:42:39 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 10 Oct 2012 22:42:39 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_updated_to_include_=22entry_point?= =?utf-8?q?=22=09functionality?= References: Message-ID: PJ Eby telecommunity.com> writes: > I'm not saying distlib must support all these plugin usecases > *itself*, but if it solves its chosen usecases in a way that can't be > *adapted by others* to handle the app platform use cases, then there's > not going to be an appealing alternative to sticking with > pkg_resources or its derivatives. Well, distlib is pretty new on the block, and it'd be interesting to get feedback from others who were trying to build on it to meet other use cases. I can't see why it couldn't be built on, but if someone came up with specific sticking points, ideally based on wanting to achieve some specific outcomes, then perhaps those points could be addressed. Since we don't even have a 0.1 release yet, there's room for discussion. Regards, Vinay Sajip From dholth at gmail.com Thu Oct 11 01:05:24 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 10 Oct 2012 19:05:24 -0400 Subject: [Distutils] distlib updated to include "entry point" functionality In-Reply-To: References: Message-ID: On Wed, Oct 10, 2012 at 6:15 PM, PJ Eby wrote: > On Tue, Oct 9, 2012 at 7:53 PM, Daniel Holth wrote: >> would also like to be able to store my installed package database in >> sqlite3 by implementing an appropriate distlib/pkg_resources backend >> and defining a standard post-(un)install "index this new package" hook >> but I doubt I will have the (unfunded) time. > > How would a database even work in that scenario (given multiple > independent directories on sys.path), and why would you expect it to > perform better than a modern filesystem? I don't know, but it would be interesting to try. For example the operation that parses every RECORD (list of installed files) to do reference counts on uninstall would probably be much faster in a database. >> Wheels are designed to be valid sys.path entries in the simple case. >> We don't encourage you to do so mostly because we don't want to >> include .pyc files for all the Pythons you might use, > > Wait... isn't wheel a binary format? Oh, I see what you mean, wheels > are a *fat* binary format, so they can include multiple Python > versions and multiple platforms. > > However, for the application platform plugins use case, wheels can > potentially be quite awesome, because you can build one fat wheel for > all your supported platforms. "We don't want to inlcude .pyc files > for all the Pythons you might use" doesn't apply to the "we" who > already build and release different platform-specific eggs to cover > multiple platforms and Python versions. ;-) If you were using wheels to implement a plugin system you could append .pyc files to the archive later, put up with slower start times, or unpack each wheel into a separate folder. A frustrating thing about eggs was that you had to replace them all even for pure-Python distributions when a new Python was released. A single wheel could mark itself as being compatible with Python 2.6-3.3, and we just let the user create the .pyc. The "fat" part has been left out of the implementation so far. The tags just mark a single wheel as working with several Pythons, usually only true for pure-Python dists. There is no provision for installing a different set of files based on the installation platform. From vinay_sajip at yahoo.co.uk Sun Oct 14 02:16:10 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 14 Oct 2012 00:16:10 +0000 (UTC) Subject: [Distutils] distlib updated with "locators" API Message-ID: I've updated distlib with an experimental API for locating distributions. A locator is just something whose "get_project" method you invoke with a "project" name such as "flask" or "sqlalchemy", and it returns a dictionary whose keys are versions of the project which were located, and the corresponding value is a tuple whose first element is a Metadata object and whose second element is a list of dicts pointing to archives for that version of the project. I've implemented three locators with some basic tests and docs: * DirectoryLocator - looks in a file system directory tree for archives * PyPIRPCLocator - uses PyPI's XML-RPC interface to locate archives * SimpleScrapingLocator - uses PyPI's "simple" web scraping interface to locate archives - similar to pip's PackageFinder A very basic script showing how to use it is at https://gist.github.com/3886402 There's plenty more work to do - if there's mileage in this idea, instead of returning Metadata and lists of dicts, Distribution instances can be returned which can be queried for dependencies in a more natural way. At the moment, this (dependency querying) isn't fully doable because all the metadata is not declarative. However, the work I've been doing with the package.yaml metadata format indicates that setup()-kwarg metadata can be automatically pulled out of setup.py for many (most) of the projects on PyPI. Feedback welcome. Regards, Vinay Sajip From dholth at gmail.com Mon Oct 15 19:32:05 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 15 Oct 2012 13:32:05 -0400 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: References: Message-ID: On Sat, Oct 13, 2012 at 8:16 PM, Vinay Sajip wrote: > There's plenty more work to do - if there's mileage in this idea, instead of > returning Metadata and lists of dicts, Distribution instances can be returned > which can be queried for dependencies in a more natural way. At the moment, > this (dependency querying) isn't fully doable because all the metadata is not > declarative. However, the work I've been doing with the package.yaml metadata > format indicates that setup()-kwarg metadata can be automatically pulled out of > setup.py for many (most) of the projects on PyPI. When you are using setuptools, setup.py can be used to generate declarative metadata with the command "python setup.py egg_info". You can add --egg-base to put the static metadata in a different directory. Setuptools even makes it easy to add new metadata files to the .egg-info folder by using a plugin to write each file. The majority of the sdists on pypi contain pre-built versions of this static metadata in .egg-info folders. Unfortunately setup.py is used to build not one, but many different dists (the most common difference of course being different requirements) depending on any number of factors, so pip rebuilds the static metadata before using it. Metadata 1.2+ of course allow us to put the most common "reasons for different dependencies" conditions in the static metadata itself. Wheel reads requires-dist lines from setup.cfg's [metadata] section so you can actually include Metadata 1.2+ style conditional dependencies. Unfortunately this feature makes it very easy to write sdists that install different dependencies if they are installed from a wheel archive versus straight from the sdist. From vinay_sajip at yahoo.co.uk Mon Oct 15 20:41:18 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 15 Oct 2012 18:41:18 +0000 (UTC) Subject: [Distutils] distlib updated with "locators" API References: Message-ID: Daniel Holth gmail.com> writes: > When you are using setuptools, setup.py can be used to generate > declarative metadata with the command "python setup.py egg_info". You > can add --egg-base to put the static metadata in a different > directory. Setuptools even makes it easy to add new metadata files to > the .egg-info folder by using a plugin to write each file. The > majority of the sdists on pypi contain pre-built versions of this > static metadata in .egg-info folders. Can it be correctly said that all of the metadata passed to setup() is in .egg-info? How far can we rely on the accuracy of the content of .egg-info dirs in sdists on PyPI? I have found hundreds of projects where even basic metadata is missing or incorrect - for example, the version. > Unfortunately setup.py is used to build not one, but many different > dists (the most common difference of course being different > requirements) depending on any number of factors, so pip rebuilds the > static metadata before using it. Metadata 1.2+ of course allow us to > put the most common "reasons for different dependencies" conditions in > the static metadata itself. Unfortunately, that doesn't help with nearly all the stuff currently on PyPI. Regards, Vinay Sajip From dholth at gmail.com Mon Oct 15 21:07:35 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 15 Oct 2012 15:07:35 -0400 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: References: Message-ID: On Mon, Oct 15, 2012 at 2:41 PM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> When you are using setuptools, setup.py can be used to generate >> declarative metadata with the command "python setup.py egg_info". You >> can add --egg-base to put the static metadata in a different >> directory. Setuptools even makes it easy to add new metadata files to >> the .egg-info folder by using a plugin to write each file. The >> majority of the sdists on pypi contain pre-built versions of this >> static metadata in .egg-info folders. > > Can it be correctly said that all of the metadata passed to setup() is in > .egg-info? How far can we rely on the accuracy of the content of .egg-info > dirs in sdists on PyPI? I have found hundreds of projects where even basic > metadata is missing or incorrect - for example, the version. You can't trust the bundled .egg-info folder, that is why pip rebuilds it during dependency resolution. You can trust the fresh one that you've just built. The .egg-info contains the three things you need during install: Name, Version, requirements. It's always been a mystery to me that we don't just edit PKG-INFO directly. >> Unfortunately setup.py is used to build not one, but many different >> dists (the most common difference of course being different >> requirements) depending on any number of factors, so pip rebuilds the >> static metadata before using it. Metadata 1.2+ of course allow us to >> put the most common "reasons for different dependencies" conditions in >> the static metadata itself. > > Unfortunately, that doesn't help with nearly all the stuff currently on PyPI. Nope, it's a ~25k-package hard problem. Or ~1.5k if you discard the simple ones. From dholth at gmail.com Tue Oct 16 20:45:57 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 16 Oct 2012 14:45:57 -0400 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: References: Message-ID: How could I have forgotten about draft PEP 390: http://www.python.org/dev/peps/pep-0390 , was looking for it in vain in the draft distutils2 docs. I don't like putting my Extensions in setup.cfg but putting at least some of the [metadata] in there is very practical. Simply write a setup.cfg for the ~1.5k packages that have conditional requirements, create a patch, and have the installer check and apply http://example.org/package/patch during the build phase. PEP 390 syntax: [metadata] name = Distribute version = 0.6.4 [metadata:sys_platform == 'win32'] requires = pywin32, bar > 1.0 obsoletes = pywin31 Current bdist_wheel syntax: [metadata] provides-extra = tool signatures faster-signatures requires-dist = distribute >= 0.6.28 markerlib argparse; python_version < '2.7' keyring; extra == 'signatures' dirspec; sys.platform != 'win32' and extra == 'signatures' ed25519ll; extra == 'faster-signatures' From merwok at netwok.org Tue Oct 16 20:52:21 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 16 Oct 2012 14:52:21 -0400 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: References: Message-ID: <507DACE5.8080101@netwok.org> Le 16/10/2012 14:45, Daniel Holth a ?crit : > How could I have forgotten about draft PEP 390: This is obsolete and should not be implemented. Regards From dholth at gmail.com Thu Oct 18 05:25:33 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 17 Oct 2012 23:25:33 -0400 Subject: [Distutils] mebs, the meta-build system Message-ID: or, removing distutils without replacing it. ** not an actual project ** Proposing mebs, the meta-build system. What if pip did not depend on setuptools or distutils and the stdlib did not include distutils or any other build system? Instead, the installer can only install binary packages, and build systems do not install but only build binary packages. A very simple meta-build system "mebs" is used to recognize sdists and build binary packages. Build systems provide plugins having three methods, .recognize() .metadata() .build() An installer downloads an sdist. For each installed build plugin, .recognize(dir) is called. The first plugin to return True is used. The plugin's .metadata(dir) is called, returning Metadata 1.3+ format metadata in a simple email.parser.Parser() multi-dict interface. Used to fetch any build/install requirements. .build(dir) is called to create the binary package. The installer installs the binary package. The first time you download an sdist, the build system/installer has to download a pre-built binary package of distutils (because it has been removed from the standard library). Alternatively a minimal setup.cfg build_system=x is consulted to download something other than distutils, the newly installed build plugin recognizes the sdist, and the package is built and installed. You wouldn't have to actually remove distutils, but every other build system could be made to compete with it on equal footing. Binary packages are used as the lingua franca because, unlike build systems, binary packages are simple enough that there can be multiple compatible implementations. From vinay_sajip at yahoo.co.uk Thu Oct 18 11:36:53 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 18 Oct 2012 09:36:53 +0000 (UTC) Subject: [Distutils] Simple API for location of distributions and requirements on PyPI Message-ID: >>> This bit of text prevents Gmane from thinking that I'm top-posting :-( I've added a convenience function to distlib.locators called locate(), to which you pass a requirement. The returned Distribution object allows you to query its requirements. Example of usage: >>> from distlib.locators import locate >>> from pprint import pprint >>> flask = locate('flask') >>> flask >>> flask.get_requirements('install') ['Werkzeug (>= 0.7)', 'Jinja2 (>= 2.4)'] >>> reqs = flask.get_requirements('install') >>> werkzeug, jinja2 = [locate(r) for r in reqs] >>> werkzeug >>> jinja2 >>> authy = locate('authy') >>> authy.get_requirements('install') ['httplib2 (>= 0.7, < 0.8)', 'simplejson'] >>> reqs = authy.get_requirements('install') >>> rds = [locate(r) for r in reqs] >>> pprint(rds) [, ] >>> Feedback welcome. Regards, Vinay Sajip From dholth at gmail.com Thu Oct 18 15:01:34 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 09:01:34 -0400 Subject: [Distutils] Simple API for location of distributions and requirements on PyPI In-Reply-To: References: Message-ID: On Thu, Oct 18, 2012 at 5:36 AM, Vinay Sajip wrote: >>>> This bit of text prevents Gmane from thinking that I'm top-posting :-( > > I've added a convenience function to distlib.locators called locate(), to which > you pass a requirement. The returned Distribution object allows you to query > its requirements. Example of usage: > >>>> from distlib.locators import locate >>>> from pprint import pprint >>>> flask = locate('flask') >>>> flask > [http://pypi.python.org/packages/source/F/Flask/Flask-0.9.tar.gz]> >>>> flask.get_requirements('install') > ['Werkzeug (>= 0.7)', 'Jinja2 (>= 2.4)'] >>>> reqs = flask.get_requirements('install') >>>> werkzeug, jinja2 = [locate(r) for r in reqs] >>>> werkzeug > [http://pypi.python.org/packages/source/W/Werkzeug/Werkzeug-0.8.3.tar.gz]> >>>> jinja2 > [http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.6.tar.gz]> >>>> authy = locate('authy') >>>> authy.get_requirements('install') > ['httplib2 (>= 0.7, < 0.8)', 'simplejson'] >>>> reqs = authy.get_requirements('install') >>>> rds = [locate(r) for r in reqs] >>>> pprint(rds) > [ [http://httplib2.googlecode.com/files/httplib2-0.7.6.tar.gz]>, > [http://pypi.python.org/packages/source/s/simplejson/simplejson-2.6.2.tar.gz]>] >>>> Rather handy. From jannis at leidel.info Thu Oct 18 19:04:02 2012 From: jannis at leidel.info (Jannis Leidel) Date: Thu, 18 Oct 2012 19:04:02 +0200 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <507DACE5.8080101@netwok.org> References: <507DACE5.8080101@netwok.org> Message-ID: <2173BA68-ACA5-4163-B0A2-D43AA65ABC5E@leidel.info> On 16.10.2012, at 20:52, ?ric Araujo wrote: > Le 16/10/2012 14:45, Daniel Holth a ?crit : >> How could I have forgotten about draft PEP 390: > This is obsolete and should not be implemented. Hi ?ric, Excuse my questions in case that information is somewhere noted but could you elaborate why it's obsolete? What's the replacement? Jannis From dholth at gmail.com Thu Oct 18 19:29:34 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 13:29:34 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 Message-ID: I'd like to submit the Wheel PEPs 425 (filename metadata), 426 (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has been stable since May and we are preparing a patch to support it in pip, but we need to earn consensus before including it in the most widely used installer. Wheel is a binary packaging format that is mostly based on the phenomenal work done by 'packaging' as PEP 376 et al. The resulting packages solve packaging problems by being installable without setup.py or any variation of distutils, lxml can be installed in 0.7 seconds, and a simple installer is just "unzip" inside site-packages. Wheel installers know about the major version number of the spec itself, and will refuse to install version 2.0 wheels if they do not understand them, so the format can be changed down the line. Let me know what I need to do to get it accepted, if anything needs to be added or revised, or, preferably, that it is awesome and you want to use it ASAP. Thanks, Daniel Holth From barry at python.org Thu Oct 18 19:34:41 2012 From: barry at python.org (Barry Warsaw) Date: Thu, 18 Oct 2012 13:34:41 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: <20121018133441.2d90634f@resist.wooz.org> On Oct 18, 2012, at 01:29 PM, Daniel Holth wrote: >Let me know what I need to do to get it accepted, if anything needs to >be added or revised, or, preferably, that it is awesome and you want >to use it ASAP. You need to advertise it on python-dev to get some wider feedback. Then ask Guido whether he wants to pronounce on it or (more likely I suspect) delegate that to a PEP czar. You should try to identify such a neutral czar in advance just in case. Cheers, -Barry From barry at python.org Thu Oct 18 19:37:52 2012 From: barry at python.org (Barry Warsaw) Date: Thu, 18 Oct 2012 13:37:52 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121018133441.2d90634f@resist.wooz.org> References: <20121018133441.2d90634f@resist.wooz.org> Message-ID: <20121018133752.7d75fecd@resist.wooz.org> On Oct 18, 2012, at 01:34 PM, Barry Warsaw wrote: >You need to advertise it on python-dev to get some wider feedback. Oops, I see you did that. Yay! -Barry From martin at v.loewis.de Thu Oct 18 19:37:11 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 18 Oct 2012 19:37:11 +0200 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> Zitat von Daniel Holth : > Let me know what I need to do to get it accepted, if anything needs to > be added or revised, or, preferably, that it is awesome and you want > to use it ASAP. As a first step, you need to find a PEP czar - somebody willing to rule on the PEP. Actually, this is the second step - at this point, you should ask for comments from the community on as many channels as you consider relevant. Many people might only now look at this for the very first time. You then need to incorporate all points you receive, even the ones you disagree with (e.g. by listing them in a discussion section, explaining why you chose to differ). Regards, Martin From dholth at gmail.com Thu Oct 18 19:47:52 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 13:47:52 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> Message-ID: On Thu, Oct 18, 2012 at 1:37 PM, wrote: > > Zitat von Daniel Holth : > >> Let me know what I need to do to get it accepted, if anything needs to >> be added or revised, or, preferably, that it is awesome and you want >> to use it ASAP. > > As a first step, you need to find a PEP czar - somebody willing to rule > on the PEP. Nick Coughlan is a likely czar. He encouraged me to write it down as a PEP in the first place. From tarek at ziade.org Thu Oct 18 19:51:20 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 18 Oct 2012 19:51:20 +0200 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> Message-ID: <50804198.2000307@ziade.org> On 10/18/12 7:47 PM, Daniel Holth wrote: > On Thu, Oct 18, 2012 at 1:37 PM, wrote: >> Zitat von Daniel Holth : >> >>> Let me know what I need to do to get it accepted, if anything needs to >>> be added or revised, or, preferably, that it is awesome and you want >>> to use it ASAP. >> As a first step, you need to find a PEP czar - somebody willing to rule >> on the PEP. > Nick Coughlan is a likely czar. He encouraged me to write it down as a > PEP in the first place. I suggest adding PEP 390 in the series of PEPs that are getting looked at. It's important to make sure they are all in sync. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From tarek at ziade.org Thu Oct 18 19:52:27 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Thu, 18 Oct 2012 19:52:27 +0200 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <507DACE5.8080101@netwok.org> References: <507DACE5.8080101@netwok.org> Message-ID: <508041DB.4050908@ziade.org> On 10/16/12 8:52 PM, ?ric Araujo wrote: > Le 16/10/2012 14:45, Daniel Holth a ?crit : >> How could I have forgotten about draft PEP 390: > This is obsolete and should not be implemented. you probably mean: the code has evolved and the PEP should reflect the latest changes > > Regards > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From dholth at gmail.com Thu Oct 18 20:00:25 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 14:00:25 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50804198.2000307@ziade.org> References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> <50804198.2000307@ziade.org> Message-ID: > I suggest adding PEP 390 in the series of PEPs that are getting looked at. > > It's important to make sure they are all in sync. Wheel archives don't include setup.cfg although PKG-INFO etc. may include information generated from setup.cfg, and bdist_wheel has its own idea of how to parse Requires-Dist PKG-INFO (taking only the parts that are not represented in setup.py) from setup.cfg. bdist_wheel is strictly a setuptools extension. The "micro language for environment markers" is unfortunately defined in multiple places. From merwok at netwok.org Thu Oct 18 20:08:41 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 18 Oct 2012 14:08:41 -0400 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <508041DB.4050908@ziade.org> References: <507DACE5.8080101@netwok.org> <508041DB.4050908@ziade.org> Message-ID: <508045A9.2080803@netwok.org> Le 18/10/2012 13:52, Tarek Ziad? a ?crit : > you probably mean: > > the code has evolved and the PEP should reflect the latest changes No, I really mean that this PEP was not developed to completion like the other free ones, was not used to guide the development of distutils2, and is no more than a historical document. You told me to ignore it two years ago. :) Cheers From tarek at fastmail.fm Thu Oct 18 20:19:21 2012 From: tarek at fastmail.fm (Tarek Ziade) Date: Thu, 18 Oct 2012 20:19:21 +0200 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <508045A9.2080803@netwok.org> References: <507DACE5.8080101@netwok.org> <508041DB.4050908@ziade.org> <508045A9.2080803@netwok.org> Message-ID: <4eaed2e3-2b93-4005-a1ce-b17797a98812@email.android.com> i dont recall this at all . its important to finish it because the syntax and the markers are a basis for several things cheers "?ric Araujo" a ?crit?: >Le 18/10/2012 13:52, Tarek Ziad? a ?crit : >> you probably mean: >> >> the code has evolved and the PEP should reflect the latest changes > >No, I really mean that this PEP was not developed to completion like >the >other free ones, was not used to guide the development of distutils2, >and is no more than a historical document. You told me to ignore it >two >years ago. :) > >Cheers >_______________________________________________ >Distutils-SIG maillist - Distutils-SIG at python.org >http://mail.python.org/mailman/listinfo/distutils-sig -- Envoy? de mon t?l?phone Android avec K-9 Mail. Excusez la bri?vet?. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Oct 18 20:24:43 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 18 Oct 2012 20:24:43 +0200 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <508041DB.4050908@ziade.org> References: <507DACE5.8080101@netwok.org> <508041DB.4050908@ziade.org> Message-ID: <20121018202443.Horde._-rUVklCcOxQgElrbjXmyxA@webmail.df.eu> Zitat von Tarek Ziad? : > On 10/16/12 8:52 PM, ?ric Araujo wrote: >> Le 16/10/2012 14:45, Daniel Holth a ?crit : >>> How could I have forgotten about draft PEP 390: >> This is obsolete and should not be implemented. > you probably mean: > > the code has evolved and the PEP should reflect the latest changes He might also have meant: - "I suggest that this PEP be rejected", or - "I'd like to ask the PEP author to withdraw the PEP" Which of these is most appropriate, I really don't know, but I agree that "is obsolete and should not be implemented" is insufficient: some action must be taken. Regards, Martin From p.f.moore at gmail.com Thu Oct 18 20:30:50 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 18 Oct 2012 19:30:50 +0100 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> <50804198.2000307@ziade.org> Message-ID: On 18 October 2012 19:00, Daniel Holth wrote: > The "micro language for environment markers" is unfortunately defined > in multiple places. Would it be worth creating a PEP covering just this aspect, then? Paul. From benjamin at python.org Thu Oct 18 20:35:19 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 18 Oct 2012 14:35:19 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: 2012/10/18 Daniel Holth : > Let me know what I need to do to get it accepted, if anything needs to > be added or revised, or, preferably, that it is awesome and you want > to use it ASAP. Traditionally, you send the peps to python-dev, so people can bikeshed inline. -- Regards, Benjamin From tarek at ziade.org Thu Oct 18 21:22:11 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Thu, 18 Oct 2012 21:22:11 +0200 Subject: [Distutils] distlib updated with "locators" API In-Reply-To: <508045A9.2080803@netwok.org> References: <507DACE5.8080101@netwok.org> <508041DB.4050908@ziade.org> <508045A9.2080803@netwok.org> Message-ID: <508056E3.9030103@ziade.org> On 10/18/12 8:08 PM, ?ric Araujo wrote: > Le 18/10/2012 13:52, Tarek Ziad? a ?crit : >> you probably mean: >> >> the code has evolved and the PEP should reflect the latest changes > No, I really mean that this PEP was not developed to completion like the > other free ones, was not used to guide the development of distutils2, > and is no more than a historical document. You told me to ignore it two > years ago. :) > > Cheers > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig Also this one looks fresher: http://alexis.notmyidea.org/distutils2/setupcfg.html and has the part about resources that's super important: http://alexis.notmyidea.org/distutils2/setupcfg.html#resources From setuptools at bugs.python.org Thu Oct 18 21:23:57 2012 From: setuptools at bugs.python.org (ranamalo) Date: Thu, 18 Oct 2012 19:23:57 +0000 Subject: [Distutils] [issue146] exclude doesn't for for find_packages Message-ID: <1350588237.75.0.97300785041.issue146@psf.upfronthosting.co.za> New submission from ranamalo: find_packages(exclude=('settings')) for examples does not exclude the file when making an egg: python setup.py bdist_egg Am I using it wrong? ---------- messages: 696 nosy: ranamalo priority: bug status: unread title: exclude doesn't for for find_packages _______________________________________________ Setuptools tracker _______________________________________________ From dholth at gmail.com Thu Oct 18 21:38:30 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 15:38:30 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121018211014.73d56057@pitrou.net> References: <20121018211014.73d56057@pitrou.net> Message-ID: On Thu, Oct 18, 2012 at 3:10 PM, Antoine Pitrou wrote: > On Thu, 18 Oct 2012 14:35:19 -0400 > Benjamin Peterson wrote: >> 2012/10/18 Daniel Holth : >> > Let me know what I need to do to get it accepted, if anything needs to >> > be added or revised, or, preferably, that it is awesome and you want >> > to use it ASAP. >> >> Traditionally, you send the peps to python-dev, so people can bikeshed inline. > > Or at least send the URLs, it's helpful. The texts seemed a bit long for direct inclusion. http://www.python.org/dev/peps/pep-0425/ http://www.python.org/dev/peps/pep-0426/ http://www.python.org/dev/peps/pep-0427/ From tarek at ziade.org Thu Oct 18 21:40:57 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 18 Oct 2012 21:40:57 +0200 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> <50804198.2000307@ziade.org> Message-ID: <50805B49.40503@ziade.org> On 10/18/12 8:30 PM, Paul Moore wrote: > On 18 October 2012 19:00, Daniel Holth wrote: >> The "micro language for environment markers" is unfortunately defined >> in multiple places. > Would it be worth creating a PEP covering just this aspect, then? > Paul. I recall I defined it there : http://www.python.org/dev/peps/pep-0345/#environment-markers From dholth at gmail.com Thu Oct 18 21:43:38 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 15:43:38 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50805B49.40503@ziade.org> References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> <50804198.2000307@ziade.org> <50805B49.40503@ziade.org> Message-ID: > I recall I defined it there : > http://www.python.org/dev/peps/pep-0345/#environment-markers Yes, that is the best place. PEP 345's supposed successor http://www.python.org/dev/peps/pep-0426/ adds ( and ) to the environment markers language, and adds the "extra" variable. From benjamin at python.org Thu Oct 18 21:44:45 2012 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 18 Oct 2012 15:44:45 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> Message-ID: 2012/10/18 Daniel Holth : > On Thu, Oct 18, 2012 at 3:10 PM, Antoine Pitrou wrote: >> On Thu, 18 Oct 2012 14:35:19 -0400 >> Benjamin Peterson wrote: >>> 2012/10/18 Daniel Holth : >>> > Let me know what I need to do to get it accepted, if anything needs to >>> > be added or revised, or, preferably, that it is awesome and you want >>> > to use it ASAP. >>> >>> Traditionally, you send the peps to python-dev, so people can bikeshed inline. >> >> Or at least send the URLs, it's helpful. > > The texts seemed a bit long for direct inclusion. Those are pretty short as PEPs go. :) -- Regards, Benjamin From pje at telecommunity.com Thu Oct 18 23:21:20 2012 From: pje at telecommunity.com (PJ Eby) Date: Thu, 18 Oct 2012 17:21:20 -0400 Subject: [Distutils] mebs, the meta-build system In-Reply-To: References: Message-ID: On Wed, Oct 17, 2012 at 11:25 PM, Daniel Holth wrote: > An installer downloads an sdist. For each installed build plugin, > .recognize(dir) is called. The first plugin to return True is used. Why not just have a standard bit of metadata in an sdist that tells the installer what builder(s) needs to be installed? i.e., the equivalent of setup_requires on steroids. (Granted, for backward compatibility, it's not hard to recognize setuptools, distribute, or distutils-based sdists.) From dholth at gmail.com Thu Oct 18 23:47:36 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 17:47:36 -0400 Subject: [Distutils] mebs, the meta-build system In-Reply-To: References: Message-ID: On Thu, Oct 18, 2012 at 5:21 PM, PJ Eby wrote: > On Wed, Oct 17, 2012 at 11:25 PM, Daniel Holth wrote: >> An installer downloads an sdist. For each installed build plugin, >> .recognize(dir) is called. The first plugin to return True is used. > > Why not just have a standard bit of metadata in an sdist that tells > the installer what builder(s) needs to be installed? i.e., the > equivalent of setup_requires on steroids. > > (Granted, for backward compatibility, it's not hard to recognize > setuptools, distribute, or distutils-based sdists.) * not an actual project * In practice you would have a tiny standard bit of metadata "setup_requires on steroids" and you would almost always follow it, but the ordered-list-of-builders design lets you override the sdist's decision. From dholth at gmail.com Fri Oct 19 05:26:59 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 18 Oct 2012 23:26:59 -0400 Subject: [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50804198.2000307@ziade.org> References: <20121018193711.Horde.zKRUP0lCcOxQgD5HXnpGUcA@webmail.df.eu> <50804198.2000307@ziade.org> Message-ID: On Thu, Oct 18, 2012 at 1:51 PM, Tarek Ziad? wrote: > On 10/18/12 7:47 PM, Daniel Holth wrote: >> >> On Thu, Oct 18, 2012 at 1:37 PM, wrote: >>> >>> Zitat von Daniel Holth : >>> >>>> Let me know what I need to do to get it accepted, if anything needs to >>>> be added or revised, or, preferably, that it is awesome and you want >>>> to use it ASAP. >>> >>> As a first step, you need to find a PEP czar - somebody willing to rule >>> on the PEP. >> >> Nick Coughlan is a likely czar. He encouraged me to write it down as a >> PEP in the first place. The wheel peps conversation continues mostly on python-dev. From jaraco at jaraco.com Sat Oct 20 10:13:56 2012 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 20 Oct 2012 08:13:56 +0000 Subject: [Distutils] Distribute 0.6.29 released Message-ID: <7E79234E600438479EC119BD241B48D623CEBEC1@CH1PRD0611MB432.namprd06.prod.outlook.com> I'm pleased to announce the release of Distribute 0.6.29. It's been far too long since the last release. This release brings a great deal of improvement , particularly around Python 3.3. Many thanks to all the people who contributed to this release. Of course, if you encounter issues, please report them at the project site . Regards, Jason R. Coombs -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 6638 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Sun Oct 21 02:01:06 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 21 Oct 2012 00:01:06 +0000 (UTC) Subject: [Distutils] semver (again) References: Message-ID: Daniel Holth gmail.com> writes: > > Taking a look at pypi, ~1k of the latest releases do not parse as > semver (if you extend it by making 1.0 equivalent to 1.0.0, as all > semver versions have at least (major, minor, patch)). This is similar > to the number of latest package versions that don't parse as PEP-386. > My latest results show that ~3600 packages fail PEP 386 versioning, with ~5400 failing semantic versioning: Packages processed: 24882 Packages with no versions: 217 Number of packages clean for all schemes: 19004 Number of packages clean for normalized (PEP 386): 21066 Number of packages clean for legacy (setuptools): 24666 Number of packages clean for semantic (semver): 19271 Where "clean" means that all versions of the packages are valid for a particular scheme. Regards, Vinay Sajip From dholth at gmail.com Sun Oct 21 04:56:49 2012 From: dholth at gmail.com (Daniel Holth) Date: Sat, 20 Oct 2012 22:56:49 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: I'm really happy with moving the troublesome Description: field to the payload. Email Policy() is useful too. http://hg.python.org/peps/rev/74868fe8ba17 Metadata 1.3 is a ... +format with no maximum line length, followed by a blank line and an +arbitrary payload. It is parseable by the ``email`` module with an +appropriate ``email.policy.Policy()``. description is deprecated... +Since Metadata 1.3 the recommended place for the description is in the +payload section of the document, after the last header. The description +needs no special formatting when included in the payload. includes a handy... # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) From tarek at ziade.org Sun Oct 21 10:32:29 2012 From: tarek at ziade.org (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 21 Oct 2012 10:32:29 +0200 Subject: [Distutils] semver (again) In-Reply-To: References: Message-ID: <5083B31D.9010405@ziade.org> On 10/21/12 2:01 AM, Vinay Sajip wrote: > Daniel Holth gmail.com> writes: > >> Taking a look at pypi, ~1k of the latest releases do not parse as >> semver (if you extend it by making 1.0 equivalent to 1.0.0, as all >> semver versions have at least (major, minor, patch)). This is similar >> to the number of latest package versions that don't parse as PEP-386. >> > My latest results show that ~3600 packages fail PEP 386 versioning, > with ~5400 failing semantic versioning: > > Packages processed: 24882 > Packages with no versions: 217 > Number of packages clean for all schemes: 19004 > Number of packages clean for normalized (PEP 386): 21066 > Number of packages clean for legacy (setuptools): 24666 > Number of packages clean for semantic (semver): 19271 > > Where "clean" means that all versions of the packages are valid for a particular > scheme. > > Regards, Have you tried to process failed version in PEP 386 with suggest_normalized_version() ? that usually raises the % considerably > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Sun Oct 21 15:17:10 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 21 Oct 2012 13:17:10 +0000 (UTC) Subject: [Distutils] semver (again) References: <5083B31D.9010405@ziade.org> Message-ID: Tarek Ziad? ziade.org> writes: > > Have you tried to process failed version in PEP 386 with > suggest_normalized_version() ? > > that usually raises the % considerably > Fair point. Including the results for "proposed" meaning normalized + suggested, I got the following: Packages processed: 24891 Packages with no versions: 217 Packages with versions: 24674 Number of packages clean for all schemes: 19010 (77%) Number of packages clean for normalized: 21072 (85%) Number of packages clean for proposed: 23685 (96%) Number of packages clean for legacy: 24674 (100%, by you would expect) Number of packages clean for semantic: 19278 (78%) Regards, Vinay Sajip From tarek at ziade.org Sun Oct 21 16:14:25 2012 From: tarek at ziade.org (=?UTF-8?B?VGFyZWsgWmlhZMOp?=) Date: Sun, 21 Oct 2012 16:14:25 +0200 Subject: [Distutils] semver (again) In-Reply-To: References: <5083B31D.9010405@ziade.org> Message-ID: <50840341.8000608@ziade.org> On 10/21/12 3:17 PM, Vinay Sajip wrote: > Tarek Ziad? ziade.org> writes: > >> Have you tried to process failed version in PEP 386 with >> suggest_normalized_version() ? >> >> that usually raises the % considerably >> > Fair point. Including the results for "proposed" meaning normalized + suggested, > I got the following: > > Packages processed: 24891 > Packages with no versions: 217 > Packages with versions: 24674 > Number of packages clean for all schemes: 19010 (77%) > Number of packages clean for normalized: 21072 (85%) > Number of packages clean for proposed: 23685 (96%) yeah that's close to what I remembered. and the 4% left are non-sense versions that said suggest_normalized_version() would probably work well with semver as well > Number of packages clean for legacy: 24674 (100%, by you would expect) > Number of packages clean for semantic: 19278 (78%) > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From vinay_sajip at yahoo.co.uk Sun Oct 21 19:01:46 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 21 Oct 2012 18:01:46 +0100 (BST) Subject: [Distutils] semver (again) In-Reply-To: References: <5083B31D.9010405@ziade.org> Message-ID: <1350838906.26773.YahooMailNeo@web171403.mail.ir2.yahoo.com> Did you mean to reply off-list? I assume it was Gmail again :-) > From: Daniel Holth >To: Vinay Sajip >Sent: Sunday, 21 October 2012, 14:33 >Subject: Re: [Distutils] semver (again) > >Of course concessions must be made to produce a practical system. Does your semver allow 1.0 in addition to 1.0.0? The SemanticVersion in distlib is currently strict and doesn't allow x.y, but the test script which produced those results did a simplistic check for matching x.y and tacked on a .0 in that case, before deciding on a pass/fail for semver. >Really what we hope to gain is a good way to order pre and post releases having the same major-minor-patch.? The "legacy" scheme (setuptools/distribute) will do that, won't it? (For some value of "good", of course.) Of course it has drawbacks, as mentioned in PEP 386: it doesn't constrain version numbers to follow a sensible scheme. Of course, to enforce any additional constraints would need PyPI metadata checks to be added, which would no doubt annoy the minority of people who name their versions like "02 Jun 2010", "pre-release", "0.1.5dev-20111031", or "0.5.0-protobuf-2.4". Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sun Oct 21 19:10:35 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 21 Oct 2012 17:10:35 +0000 (UTC) Subject: [Distutils] semver (again) References: <5083B31D.9010405@ziade.org> <50840341.8000608@ziade.org> Message-ID: Tarek Ziad? ziade.org> writes: > yeah that's close to what I remembered. and the 4% left are non-sense > versions > > that said suggest_normalized_version() would probably work well with > semver as well > Actually, sensible-looking semantic versions seem not to be amenable to having normalized versions suggested via suggest_normalized_version(). For example, the legal semantic version "0.1.0-build+post" fails both as-is, and for suggestion: >>> from distutils2 import version >>> version.NormalizedVersion('0.1.0-build+post') Traceback (most recent call last): File "", line 1, in File "distutils2/version.py", line 84, in __init__ self._parse(s, error_on_huge_major_num) File "distutils2/version.py", line 95, in _parse raise IrrationalVersionError(s) distutils2.errors.IrrationalVersionError: 0.1.0-build+post >>> assert version.suggest_normalized_version('0.1.0-build+post') Traceback (most recent call last): File "", line 1, in AssertionError So, some work might be required for NormalizedVersion to work with semantic versions. Regards, Vinay Sajip From techtonik at gmail.com Mon Oct 22 15:06:03 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 22 Oct 2012 16:06:03 +0300 Subject: [Distutils] distribute buildbot In-Reply-To: <503DF240.2010201@gmail.com> References: <50374470.40105@gmail.com> <503DF240.2010201@gmail.com> Message-ID: On Wed, Aug 29, 2012 at 1:43 PM, Adam GROSZER wrote: > To answer myself, winbot.zooe.org builds now distribute The link is dead. Where is it now? From techtonik at gmail.com Mon Oct 22 15:12:01 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 22 Oct 2012 16:12:01 +0300 Subject: [Distutils] Distribute + Sphinx Fail in Python 3.2/3.3 on Windows Message-ID: I can't install Sphinx with pristine installation of Python 3.2 and Python 3.3 on Windows and it looks like there some bugs in distribute_setup.py and Distribute versions. Can anybody take a look? I think it's rather critical. https://bitbucket.org/birkenfeld/sphinx/issue/1022/doesnt-install-with-python-32-and-33-on If it is bug in Distribute, maybe distribute_setup.py should be patched to check for known issues for default version specified inside and upgrade itself instead of giving users bad times? -- anatoly t. From techtonik at gmail.com Mon Oct 22 20:03:48 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 22 Oct 2012 21:03:48 +0300 Subject: [Distutils] __build__ as a temp build directory for setup.py Message-ID: What do you think about this? http://bugs.python.org/issue16299 From techtonik at gmail.com Mon Oct 22 21:57:52 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 22 Oct 2012 22:57:52 +0300 Subject: [Distutils] How to upgrade distribute with distribute_setup if necessary? Message-ID: I can't install Sphinx in a Python 3.2 directory with distribute 0.6.13 installed due to a bug in outdated distribute. Sphinx uses distribute_setup.py and if no setuptools installed it fetches new version where there is no such problem. The check is the following code: try: from setuptools import setup, find_packages except ImportError: import distribute_setup distribute_setup.use_setuptools() from setuptools import setup, find_packages What is the proper way to modify it to make distribute_setup.py upgrade distribute if necessary? -- anatoly t. From dholth at gmail.com Mon Oct 22 22:00:35 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 22 Oct 2012 16:00:35 -0400 Subject: [Distutils] How to upgrade distribute with distribute_setup if necessary? In-Reply-To: References: Message-ID: On Mon, Oct 22, 2012 at 3:57 PM, anatoly techtonik wrote: > I can't install Sphinx in a Python 3.2 directory with distribute > 0.6.13 installed due to a bug in outdated distribute. Sphinx uses > distribute_setup.py and if no setuptools installed it fetches new > version where there is no such problem. The check is the following > code: > > try: > from setuptools import setup, find_packages > except ImportError: > import distribute_setup > distribute_setup.use_setuptools() > from setuptools import setup, find_packages > > What is the proper way to modify it to make distribute_setup.py > upgrade distribute if necessary? > -- > anatoly t. You should just invoke a recent virtualenv on the environment. From techtonik at gmail.com Mon Oct 22 22:08:45 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 22 Oct 2012 23:08:45 +0300 Subject: [Distutils] How to upgrade distribute with distribute_setup if necessary? In-Reply-To: References: Message-ID: On Mon, Oct 22, 2012 at 11:00 PM, Daniel Holth wrote: > On Mon, Oct 22, 2012 at 3:57 PM, anatoly techtonik wrote: >> I can't install Sphinx in a Python 3.2 directory with distribute >> 0.6.13 installed due to a bug in outdated distribute. Sphinx uses >> distribute_setup.py and if no setuptools installed it fetches new >> version where there is no such problem. The check is the following >> code: >> >> try: >> from setuptools import setup, find_packages >> except ImportError: >> import distribute_setup >> distribute_setup.use_setuptools() >> from setuptools import setup, find_packages >> >> What is the proper way to modify it to make distribute_setup.py >> upgrade distribute if necessary? >> -- >> anatoly t. > > You should just invoke a recent virtualenv on the environment. It is for system-wide installation of Sphinx. I need it as a user tool. -- anatoly t. From vinay_sajip at yahoo.co.uk Mon Oct 22 23:40:00 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 22 Oct 2012 21:40:00 +0000 (UTC) Subject: [Distutils] =?utf-8?q?How_to_upgrade_distribute_with_distribute?= =?utf-8?q?=5Fsetupif=09necessary=3F?= References: Message-ID: anatoly techtonik gmail.com> writes: > > On Mon, Oct 22, 2012 at 11:00 PM, Daniel Holth gmail.com> wrote: > > You should just invoke a recent virtualenv on the environment. > > It is for system-wide installation of Sphinx. I need it as a user tool. > -- Download the latest version of distribute_setup.py from http://python-distribute.org/distribute_setup.py Then run it (using sudo if necessary). It should download the latest distribute version (0.6.30) and install it in your environment. You might want to try it in a virtualenv first. Regards, Vinay Sajip From dholth at gmail.com Mon Oct 22 23:49:12 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 22 Oct 2012 17:49:12 -0400 Subject: [Distutils] How to upgrade distribute with distribute_setupif necessary? In-Reply-To: References: Message-ID: If this system uses rpm or deb it will be beneficial to find a newer system package of distribute if at all possible. On Oct 22, 2012 5:44 PM, "Vinay Sajip" wrote: > anatoly techtonik gmail.com> writes: > > > > > On Mon, Oct 22, 2012 at 11:00 PM, Daniel Holth gmail.com> > wrote: > > > You should just invoke a recent virtualenv on the environment. > > > > It is for system-wide installation of Sphinx. I need it as a user tool. > > -- > > Download the latest version of distribute_setup.py from > > http://python-distribute.org/distribute_setup.py > > Then run it (using sudo if necessary). It should download the latest > distribute > version (0.6.30) and install it in your environment. You might want to try > it in > a virtualenv first. > > Regards, > > Vinay Sajip > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Tue Oct 23 00:21:51 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Tue, 23 Oct 2012 00:21:51 +0200 Subject: [Distutils] __build__ as a temp build directory for setup.py In-Reply-To: References: Message-ID: <20121023002151.Horde.AQsVDVNNcXdQhcb-v4RhCIA@webmail.df.eu> Zitat von anatoly techtonik : > What do you think about this? > > http://bugs.python.org/issue16299 I think the status quo is fine, and see no reason to change it. Regards, Martin From ronaldoussoren at mac.com Tue Oct 23 08:28:47 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 23 Oct 2012 08:28:47 +0200 Subject: [Distutils] __build__ as a temp build directory for setup.py In-Reply-To: References: Message-ID: On 22 Oct, 2012, at 20:03, anatoly techtonik wrote: > What do you think about this? > > http://bugs.python.org/issue16299 The cost of changing the build directory is high, and has limited upsides at best. Some of the costs: confusing current users, breaking existing documentation like books, breaking build systems, incompatibility between python versions. If the name of the build directory bothers you you can change it by adding two lines to ~/.pydistutils.cfg: [build] build-base = __build__ Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig From ronaldoussoren at mac.com Wed Oct 24 13:04:36 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 24 Oct 2012 13:04:36 +0200 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> On 18 Oct, 2012, at 19:29, Daniel Holth wrote: > I'd like to submit the Wheel PEPs 425 (filename metadata), 426 > (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has > been stable since May and we are preparing a patch to support it in > pip, but we need to earn consensus before including it in the most > widely used installer. PEP 425: * "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead" I don't particularly like replacing dots by underscores. That needed because you use the dot character in compressed tag sets, but why not use a comma to separate items in the compressed tag set? * "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." Why the replacement? The need for replacement could be avoided by using a different separator between elements of a tag (for example "~" or "+"), and furthermore the platform tag is at a know location, and hence the use of hyphens in the platform tag is harmless (use "python_tag, abi_tag, platform_tag = tag.split('-', 2)" to split the tag into its elements. * "compressed tag sets" Using '"," instead of "." to separate elements of the tag set takes away the need to replace dots in tag elements, and seems more natural to me (you'd also use comma to separate the elements when you write them down in prose or python code. Ronald From ronaldoussoren at mac.com Wed Oct 24 13:28:48 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 24 Oct 2012 13:28:48 +0200 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: On 18 Oct, 2012, at 19:29, Daniel Holth wrote: > I'd like to submit the Wheel PEPs 425 (filename metadata), 426 > (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has > been stable since May and we are preparing a patch to support it in > pip, but we need to earn consensus before including it in the most > widely used installer. PEP 427: * The installation section mentions that .py files should be compiled to .pyc/.pyo files, and that "Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.". Wouldn't it be better to add the compiled files to the RECORD file? That would break the digital signature, but I'm not sure if verifying the signature post-installation is useful (or if it's even intended to work). * Why is urlsafe_b64encode_nopad used to encode the hash in the record file, instead of the normal hex encoding that's directly supported by the hash module and system tools? * The way to specify the required public key in package requirements in ugly (it looks like an abuse of setuptools' extras mechanism). Is there really no nicer way to specify this? * As was noted before there is no threat model for the signature feature, which makes it hard to evaluate if the feature. In particular, what is the advantage of this over PGP signatures of wheels? (PyPI already supports detached signatures, and such signatures are used more widely in the OSS world) * RECORD.p7s is not described at all. I'm assuming this is intented to be a X.509 signature of RECORD in pkcs7 format. Why PKCS7 and not PEM? The latter seems to be easier to work with. Ronald From dholth at gmail.com Wed Oct 24 14:48:11 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 24 Oct 2012 08:48:11 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: On Wed, Oct 24, 2012 at 7:28 AM, Ronald Oussoren wrote: > > On 18 Oct, 2012, at 19:29, Daniel Holth wrote: > >> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >> been stable since May and we are preparing a patch to support it in >> pip, but we need to earn consensus before including it in the most >> widely used installer. > > PEP 427: > > * The installation section mentions that .py files should be compiled to .pyc/.pyo files, and that "Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.". > > Wouldn't it be better to add the compiled files to the RECORD file? That would break the digital signature, but I'm not sure if verifying the signature post-installation is useful (or if it's even > intended to work). The trouble with mentioning .pyc files in RECORD is that someone can install Python 3.4, and suddenly you have additional .pyc files, approximately __pycache__/pyfile.cp34.pyc. So you should remove more than what you installed anyway. You can't verify the signature post-installation. #!python and RECORD have been rewritten at this point. > * Why is urlsafe_b64encode_nopad used to encode the hash in the record file, instead of the normal hex encoding that's directly supported by the hash module and system tools? It's nice and small. The encoder is just base64.urlsafe_b64encode(digest).rstrip('=') > * The way to specify the required public key in package requirements in ugly (it looks like an abuse of setuptools' extras mechanism). Is there really no nicer way to specify this? > > * As was noted before there is no threat model for the signature feature, which makes it hard to evaluate if the feature. In particular, what is the advantage of this over PGP signatures of wheels? (PyPI already supports detached signatures, and such signatures are used more widely in the OSS world) > > * RECORD.p7s is not described at all. I'm assuming this is intented to be a X.509 signature of RECORD in pkcs7 format. Why PKCS7 and not PEM? The latter seems to be easier to work with. I am very confused about the idea that not-downloading-the-archive-you-expected (pypi accounts getting hacked, man-in-the-middle attacks, simply using the wrong index) is an unrealistic threat. It might help to think of the wheel signing scheme as a more powerful version of the current #md5=digest instead of comparing it to PGP or TLS. An md5 sum verifies the integrity of a single archive, the wheel signing key verifies the integrity of any number of archives. Like the archive digest, wheel just explains how to attach the signature to the archive. A system for [automatically] trusting any particular key could be built separately. Wheel's signing scheme is similar to jarsigner. The big advantage over PGP is that they are attached and less likely to get lost. PyPI still supports detached signatures, even on wheel files, but they are unpopular. Wheel gives you an additional different option. Since the signature is over the unpacked contents, you can also change the compression algorithm in the zipfile or append another signature without invalidating the existing signature. The simplified certificate model is inspired by SPKI/SDSI (http://world.std.com/~cme/html/spki.html), Convergence (http://convergence.io/) TACK (http://tack.io), and the general discussion about the brokenness of the certificate authority system. You get the raw public key without a claim that it represents anything or anyone. PKCS7 is the format that a US government user would be required to use with their smartcard-based system. I like the packagename[algorithm=key] syntax even though it started as a hack. It fits into the existing pip requirements.txt syntax perfectly, unlike packagename[extra]#algorithm=key, and it reads like array indexing. From dholth at gmail.com Wed Oct 24 14:59:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 24 Oct 2012 08:59:31 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> References: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> Message-ID: On Wed, Oct 24, 2012 at 7:04 AM, Ronald Oussoren wrote: > > On 18 Oct, 2012, at 19:29, Daniel Holth wrote: > >> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >> been stable since May and we are preparing a patch to support it in >> pip, but we need to earn consensus before including it in the most >> widely used installer. > > PEP 425: > > * "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead" > > I don't particularly like replacing dots by underscores. That needed because you use the dot character in compressed tag sets, but why not use a comma to separate items in the compressed tag set? > * "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." > > Why the replacement? The need for replacement could be avoided by using a different separator between elements of a tag (for example "~" or "+"), and furthermore the platform tag is at a know > location, and hence the use of hyphens in the platform tag is harmless (use "python_tag, abi_tag, platform_tag = tag.split('-', 2)" to split the tag into its elements. This is based on the longstanding convention of folding - and _ (hyphen and underscore) in built distribution filenames and using - to separate parts. > * "compressed tag sets" > > Using '"," instead of "." to separate elements of the tag set takes away the need to replace dots in tag elements, and seems more natural to me (you'd also use comma to separate the elements > when you write them down in prose or python code. I kindof like the , The + might transform into a space in URLs? From techtonik at gmail.com Thu Oct 25 09:23:54 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 25 Oct 2012 10:23:54 +0300 Subject: [Distutils] __build__ as a temp build directory for setup.py In-Reply-To: References: Message-ID: On Tue, Oct 23, 2012 at 9:28 AM, Ronald Oussoren wrote: > > On 22 Oct, 2012, at 20:03, anatoly techtonik wrote: > >> What do you think about this? >> >> http://bugs.python.org/issue16299 > > The cost of changing the build directory is high, and has limited upsides at best. Some of the costs: confusing current users, breaking existing documentation like books, breaking build systems, incompatibility between python versions. There is already a lot of incompatibilities between 2 and 3, and even between 3.1, 3.2 and 3.3, so the added value of this change to the total cost is miserable. Speaking about user confusion - if they won't find 'build' directory - they'll surely notice __build__ in their tree. Python 3 toolchains are still fragile, so it won't come for me a a surprise if Python 3.4 conventions are not the same as in 3.3. So, there are two questions: 1. If you were designing Python from scratch right now now - which name would you choose `__build__` or `build` for the temporary directory? 2. Is the Python 3.x already mature enough to deny any improvements (considering these are improvements) for the 3.4 version? > If the name of the build directory bothers you you can change it by adding two lines to ~/.pydistutils.cfg: > > [build] > build-base = __build__ From andrew.svetlov at gmail.com Thu Oct 25 10:29:06 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 25 Oct 2012 11:29:06 +0300 Subject: [Distutils] __build__ as a temp build directory for setup.py In-Reply-To: References: Message-ID: I think renaming doesn't makes big value, so better to stay with current state. On Thu, Oct 25, 2012 at 10:23 AM, anatoly techtonik wrote: > On Tue, Oct 23, 2012 at 9:28 AM, Ronald Oussoren wrote: >> >> On 22 Oct, 2012, at 20:03, anatoly techtonik wrote: >> >>> What do you think about this? >>> >>> http://bugs.python.org/issue16299 >> >> The cost of changing the build directory is high, and has limited upsides at best. Some of the costs: confusing current users, breaking existing documentation like books, breaking build systems, incompatibility between python versions. > > There is already a lot of incompatibilities between 2 and 3, and even > between 3.1, 3.2 and 3.3, so the added value of this change to the > total cost is miserable. Speaking about user confusion - if they won't > find 'build' directory - they'll surely notice __build__ in their > tree. Python 3 toolchains are still fragile, so it won't come for me a > a surprise if Python 3.4 conventions are not the same as in 3.3. > > So, there are two questions: > 1. If you were designing Python from scratch right now now - which > name would you choose `__build__` or `build` for the temporary > directory? > 2. Is the Python 3.x already mature enough to deny any improvements > (considering these are improvements) for the 3.4 version? > >> If the name of the build directory bothers you you can change it by adding two lines to ~/.pydistutils.cfg: >> >> [build] >> build-base = __build__ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig -- Thanks, Andrew Svetlov From ronaldoussoren at mac.com Thu Oct 25 11:07:13 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 25 Oct 2012 11:07:13 +0200 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> Message-ID: On 24 Oct, 2012, at 14:59, Daniel Holth wrote: > On Wed, Oct 24, 2012 at 7:04 AM, Ronald Oussoren wrote: >> >> On 18 Oct, 2012, at 19:29, Daniel Holth wrote: >> >>> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >>> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >>> been stable since May and we are preparing a patch to support it in >>> pip, but we need to earn consensus before including it in the most >>> widely used installer. >> >> PEP 425: >> >> * "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead" >> >> I don't particularly like replacing dots by underscores. That needed because you use the dot character in compressed tag sets, but why not use a comma to separate items in the compressed tag set? > >> * "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." >> >> Why the replacement? The need for replacement could be avoided by using a different separator between elements of a tag (for example "~" or "+"), and furthermore the platform tag is at a know >> location, and hence the use of hyphens in the platform tag is harmless (use "python_tag, abi_tag, platform_tag = tag.split('-', 2)" to split the tag into its elements. > > This is based on the longstanding convention of folding - and _ > (hyphen and underscore) in built distribution filenames and using - to > separate parts. AFAIK distutils and setuptools do not replace hyphens in the platform name in the name of bdist files. > >> * "compressed tag sets" >> >> Using '"," instead of "." to separate elements of the tag set takes away the need to replace dots in tag elements, and seems more natural to me (you'd also use comma to separate the elements >> when you write them down in prose or python code. > > I kindof like the , > > The + might transform into a space in URLs? You're right, + is not a good choice because that character must be quoted in URLs. Ronald From vinay_sajip at yahoo.co.uk Thu Oct 25 11:38:20 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 25 Oct 2012 09:38:20 +0000 (UTC) Subject: [Distutils] distlib locator compares reasonably well with pip's PackageFinder Message-ID: As an extended test of the locator functionality in distlib, I compared the results from the locate() API with the results from pip's PackageFinder. The results seem encouraging: out of 24938 packages tested, only 510 (2%) gave different results from pip. Out of the 510, many are due to: 1. distlib (currently) looks for archives which are machine-independent, and skips archives which contain machine-dependent strings in their names. 2. distlib (currently) skips archives automatically generated by sites like GitHub and BitBucket, since the download archive doesn't always contain version information in the name of the download. Examples: http://github.com/user/project/tarball/master http://bitbucket.org/user/project/get/tip.zip In a (much) smaller number of cases, the differences are because locate() found an archive where pip didn't. But for 98% of the projects registered on PyPI, locate() returns identical results to pip's PackageFinder.find_requirement, and does it slightly faster in most cases :-) The comparison script, along with the list of differences, is at https://gist.github.com/3951634 Regards, Vinay Sajip From ronaldoussoren at mac.com Thu Oct 25 10:46:16 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 25 Oct 2012 10:46:16 +0200 Subject: [Distutils] __build__ as a temp build directory for setup.py In-Reply-To: References: Message-ID: On 25 Oct, 2012, at 9:23, anatoly techtonik wrote: > On Tue, Oct 23, 2012 at 9:28 AM, Ronald Oussoren wrote: >> >> On 22 Oct, 2012, at 20:03, anatoly techtonik wrote: >> >>> What do you think about this? >>> >>> http://bugs.python.org/issue16299 >> >> The cost of changing the build directory is high, and has limited upsides at best. Some of the costs: confusing current users, breaking existing documentation like books, breaking build systems, incompatibility between python versions. > > There is already a lot of incompatibilities between 2 and 3, and even > between 3.1, 3.2 and 3.3, so the added value of this change to the > total cost is miserable. Speaking about user confusion - if they won't > find 'build' directory - they'll surely notice __build__ in their > tree. Python 3 toolchains are still fragile, so it won't come for me a > a surprise if Python 3.4 conventions are not the same as in 3.3. What do you mean w.r.t. python 3 toolchain fragility? AFAIK distutils and distribute work fine with Python 3, and have the same interface as with Python 2. > > So, there are two questions: > 1. If you were designing Python from scratch right now now - which > name would you choose `__build__` or `build` for the temporary > directory? I don't know what I'd choose, __build__ is a bit too cute for my taste. I do know that I also use "build" as the name for build directories outside of Python, and have done so from before I used Python. Anyway, we are not designing from scratch, and that means you have to take the cost of change into account. Given the, at best, small improvement I don't think it worthwhile to change the default. See also . > 2. Is the Python 3.x already mature enough to deny any improvements > (considering these are improvements) for the 3.4 version? Of course not, but there is a balance between the cost of changes and the improvements those bring. Ronald > >> If the name of the build directory bothers you you can change it by adding two lines to ~/.pydistutils.cfg: >> >> [build] >> build-base = __build__ From ronaldoussoren at mac.com Fri Oct 26 09:54:00 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 26 Oct 2012 09:54:00 +0200 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: Message-ID: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> On 24 Oct, 2012, at 14:48, Daniel Holth wrote: > On Wed, Oct 24, 2012 at 7:28 AM, Ronald Oussoren wrote: >> >> On 18 Oct, 2012, at 19:29, Daniel Holth wrote: >> >>> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >>> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >>> been stable since May and we are preparing a patch to support it in >>> pip, but we need to earn consensus before including it in the most >>> widely used installer. >> >> PEP 427: >> >> * The installation section mentions that .py files should be compiled to .pyc/.pyo files, and that "Uninstallers should be smart enough to remove .pyc even if it is not mentioned in RECORD.". >> >> Wouldn't it be better to add the compiled files to the RECORD file? That would break the digital signature, but I'm not sure if verifying the signature post-installation is useful (or if it's even >> intended to work). > > The trouble with mentioning .pyc files in RECORD is that someone can > install Python 3.4, and suddenly you have additional .pyc files, > approximately __pycache__/pyfile.cp34.pyc. So you should remove more > than what you installed anyway. > > You can't verify the signature post-installation. #!python and RECORD > have been rewritten at this point. > >> * Why is urlsafe_b64encode_nopad used to encode the hash in the record file, instead of the normal hex encoding that's directly supported by the hash module and system tools? > > It's nice and small. The encoder is just > base64.urlsafe_b64encode(digest).rstrip('=') But is the size difference really important? The wheel file itself is compressed, and the additional amount of space needed on installation shouldn't be a problem. The advantage of using hexdigest is that both the "classic" MD5 checksum and the new tagged checksums you propose then use the same encoding for the signature. > >> * The way to specify the required public key in package requirements in ugly (it looks like an abuse of setuptools' extras mechanism). Is there really no nicer way to specify this? >> >> * As was noted before there is no threat model for the signature feature, which makes it hard to evaluate if the feature. In particular, what is the advantage of this over PGP signatures of wheels? (PyPI already supports detached signatures, and such signatures are used more widely in the OSS world) >> >> * RECORD.p7s is not described at all. I'm assuming this is intented to be a X.509 signature of RECORD in pkcs7 format. Why PKCS7 and not PEM? The latter seems to be easier to work with. > > I am very confused about the idea that > not-downloading-the-archive-you-expected (pypi accounts getting > hacked, man-in-the-middle attacks, simply using the wrong index) is an > unrealistic threat. You don't mention the threats you try to protect against in the PEP. Users are still somewhat vulnerable to the attacks to mention when the download new software, they still have to get the public key somewhere. In the example of using a requirements.txt file with public keys I'd still have to get that file from somewhere and maybe that location was attacked. > > It might help to think of the wheel signing scheme as a more powerful > version of the current #md5=digest instead of comparing it to PGP or > TLS. An md5 sum verifies the integrity of a single archive, the wheel > signing key verifies the integrity of any number of archives. Like the > archive digest, wheel just explains how to attach the signature to the > archive. A system for [automatically] trusting any particular key > could be built separately. > > Wheel's signing scheme is similar to jarsigner. The big advantage over > PGP is that they are attached and less likely to get lost. PyPI still > supports detached signatures, even on wheel files, but they are > unpopular. Wheel gives you an additional different option. RPM uses embedded PGP signatures, and those are easy enough to use. PGP signatures on PyPI require a PGP installation on the users machine, your scheme at least has the advantage of not needing additional software. > > Since the signature is over the unpacked contents, you can also change > the compression algorithm in the zipfile or append another signature > without invalidating the existing signature. > > The simplified certificate model is inspired by SPKI/SDSI > (http://world.std.com/~cme/html/spki.html), Convergence > (http://convergence.io/) TACK (http://tack.io), and the general > discussion about the brokenness of the certificate authority system. I've added these to my reading list. I know just enough of crypto/signatures to be dangerous, which might explain why I worry about something that isn't old and used a lot. > You get the raw public key without a claim that it represents anything > or anyone. Simularly to the CA system :-) > > PKCS7 is the format that a US government user would be required to use > with their smartcard-based system. > > I like the packagename[algorithm=key] syntax even though it started as > a hack. It fits into the existing pip requirements.txt syntax > perfectly, unlike packagename[extra]#algorithm=key, and it reads like > array indexing. I don't like the extras hack, but don't have a better solution. Ronald From p.f.moore at gmail.com Fri Oct 26 10:11:50 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 26 Oct 2012 09:11:50 +0100 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> References: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> Message-ID: On 26 October 2012 08:54, Ronald Oussoren wrote: >> >> It's nice and small. The encoder is just >> base64.urlsafe_b64encode(digest).rstrip('=') > > But is the size difference really important? The wheel file itself is compressed, and the additional > amount of space needed on installation shouldn't be a problem. The advantage of using hexdigest > is that both the "classic" MD5 checksum and the new tagged checksums you propose then use > the same encoding for the signature. I agree. This encoding seems to be a micro-optimisation with no real justification. I'd prefer to see hexdigest used, as (a) it means md5 is not a special case, and (b) there's not a proliferation of 1-line functions in use code doing that b64encode/strip dance. With hexdigest, the syntax is just [algorithm=]hexdigest, where algorithm defaults to md5. Much simpler to describe and work with. Paul. From dholth at gmail.com Fri Oct 26 18:28:28 2012 From: dholth at gmail.com (Daniel Holth) Date: Fri, 26 Oct 2012 12:28:28 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <74DF21FC-447C-4CB0-A55B-D8CC839B165D@mac.com> Message-ID: On Fri, Oct 26, 2012 at 4:11 AM, Paul Moore wrote: > On 26 October 2012 08:54, Ronald Oussoren wrote: >>> >>> It's nice and small. The encoder is just >>> base64.urlsafe_b64encode(digest).rstrip('=') >> >> But is the size difference really important? The wheel file itself is compressed, and the additional >> amount of space needed on installation shouldn't be a problem. The advantage of using hexdigest >> is that both the "classic" MD5 checksum and the new tagged checksums you propose then use >> the same encoding for the signature. > > I agree. This encoding seems to be a micro-optimisation with no real > justification. I'd prefer to see hexdigest used, as (a) it means md5 > is not a special case, and (b) there's not a proliferation of 1-line > functions in use code doing that b64encode/strip dance. > > With hexdigest, the syntax is just [algorithm=]hexdigest, where > algorithm defaults to md5. Much simpler to describe and work with. Why don't we get just get rid of the backwards-compatible (with a previous version of PEP 376) "hexdigest defaults to md5" syntax, and require name=base64 digest. Is there any code that parses the PEP 376 digest? base64 is not hard. I was a little surprised that the optional-padding versions were not already in the standard library. def urlsafe_b64encode(data): return base64.urlsafe_b64encode(data).rstrip(binary('=')) def urlsafe_b64decode(data): pad = b'=' * (4 - (len(data) & 3)) return base64.urlsafe_b64decode(data + pad) From ralf at systemexit.de Fri Oct 26 20:53:00 2012 From: ralf at systemexit.de (Ralf Schmitt) Date: Fri, 26 Oct 2012 20:53:00 +0200 Subject: [Distutils] [ANN] pypiserver 1.0.0 - minimal private pypi server Message-ID: <87625xjcab.fsf@myhost.lan> Hi, I've just uploaded pypiserver 1.0.0 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. http://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in version 1.0.0 ------------------------ - add passlib and waitress to pypi-server-standalone - upgrade bottle to 0.11.3 - Update scripts/opensuse/pypiserver.init - Refuse to re upload existing file - Add 'console_scripts' section to 'entry_points', so 'pypi-server.exe' will be created on Windows. - paste_app_factory now use the the password_file option to create the app. Without this the package upload was not working. - Add --fallback-url argument to pypi-server script to make it configurable. -- Cheers Ralf From aclark at aclark.net Fri Oct 26 21:41:55 2012 From: aclark at aclark.net (Alex Clark) Date: Fri, 26 Oct 2012 15:41:55 -0400 Subject: [Distutils] [ANN] pypiserver 1.0.0 - minimal private pypi server References: <87625xjcab.fsf@myhost.lan> Message-ID: Hi Ralf, On 2012-10-26 18:53:00 +0000, Ralf Schmitt said: > Hi, > > I've just uploaded pypiserver 1.0.0 to the python package index. > > pypiserver is a minimal PyPI compatible server. It can be used to serve > a set of packages and eggs to easy_install or pip. > > pypiserver is easy to install (i.e. just 'pip install pypiserver'). It > doesn't have any external dependencies. > > http://pypi.python.org/pypi/pypiserver/ should contain enough > information to easily get you started running your own PyPI server in a > few minutes. > > The code is available on github: https://github.com/schmir/pypiserver > > Changes in version 1.0.0 > ------------------------ > - add passlib and waitress to pypi-server-standalone > - upgrade bottle to 0.11.3 > - Update scripts/opensuse/pypiserver.init > - Refuse to re upload existing file > - Add 'console_scripts' section to 'entry_points', so > 'pypi-server.exe' will be created on Windows. > - paste_app_factory now use the the password_file option to create the > app. Without this the package upload was not working. > - Add --fallback-url argument to pypi-server script to make it > configurable. Very cool! Thank you -- Alex Clark ? https://www.gittip.com/aclark4life/ From pje at telecommunity.com Sat Oct 27 03:45:35 2012 From: pje at telecommunity.com (PJ Eby) Date: Fri, 26 Oct 2012 21:45:35 -0400 Subject: [Distutils] distlib locator compares reasonably well with pip's PackageFinder In-Reply-To: References: Message-ID: On Thu, Oct 25, 2012 at 5:38 AM, Vinay Sajip wrote: > 2. distlib (currently) skips archives automatically generated by sites like > GitHub and BitBucket, since the download archive doesn't always contain > version information in the name of the download. Examples: > > http://github.com/user/project/tarball/master > http://bitbucket.org/user/project/get/tip.zip Just as an FYI, are you aware of the #egg=projname-version tagging convention currently in use for such links? From vinay_sajip at yahoo.co.uk Sat Oct 27 11:57:02 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 27 Oct 2012 09:57:02 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_locator_compares_reasonably_well_wi?= =?utf-8?q?th_pip=27s=09PackageFinder?= References: Message-ID: PJ Eby telecommunity.com> writes: > > Just as an FYI, are you aware of the #egg=projname-version tagging > convention currently in use for such links? I wasn't - thanks - still new to this game :-) In the cases I was referring to, the fragment looks like #egg=projname-dev. I still have to look through the 2% where distlib's behaviour is different and check what the reason is. I'm not sure if we should, as default behaviour, identify such archives as potential downloads, especially as a goal is to do dependency resolution without having to actually download archives and inspecting PKG-INFO / running egg-info on them etc. Of course, this verification could be done during actual installation. Regards, Vinay Sajip From pje at telecommunity.com Sat Oct 27 21:53:41 2012 From: pje at telecommunity.com (PJ Eby) Date: Sat, 27 Oct 2012 15:53:41 -0400 Subject: [Distutils] distlib locator compares reasonably well with pip's PackageFinder In-Reply-To: References: Message-ID: On Sat, Oct 27, 2012 at 5:57 AM, Vinay Sajip wrote: > PJ Eby telecommunity.com> writes: > >> >> Just as an FYI, are you aware of the #egg=projname-version tagging >> convention currently in use for such links? > > I wasn't - thanks - still new to this game :-) In the cases I was referring to, > the fragment looks like #egg=projname-dev. "dev" is the version, actually. It's a perfectly valid version to setuptools, and parses as a version that's below any commonly-used version. This lets people specify "==dev" to target an in-development version for installation -- usually manually, but sometimes automatically. One might specify, for example "foobar>2.0,==dev" to tell setuptools that if you can't find a released version >2.0, then an in-development version is acceptable. > I'm not sure if we should, as default behaviour, identify such archives as > potential downloads, If they're using a "dev" version, then such a link is automatically lower-precedence than anything else already, due to it being the lowest available version.(Newer tools could treat it as 0dev or whatever the official translation/suggestion is.) In addition, it denotes a "non-stable" version, so if the tool allows one to prioritize stable versions, it'll be eliminated as a candidate anyway in that case. If the version tag is precise, OTOH, (i.e., something other than 'dev'), then presumably the provider of the link can be trusted to have identified what version it is. IIUC, those source control sites let you download tarballs of arbitrary versions, so one could in principle issue download releases of exact source snapshots. (Indeed, it's not a bad way to go about it.) From vinay_sajip at yahoo.co.uk Sun Oct 28 01:08:27 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 27 Oct 2012 23:08:27 +0000 (UTC) Subject: [Distutils] =?utf-8?q?distlib_locator_compares_reasonably_well_wi?= =?utf-8?q?th_pip=27s=09PackageFinder?= References: Message-ID: PJ Eby telecommunity.com> writes: > "dev" is the version, actually. It's a perfectly valid version to > setuptools, and parses as a version that's below any commonly-used > version. This lets people specify "==dev" to target an in-development > version for installation -- usually manually, but sometimes > automatically. One might specify, for example "foobar>2.0,==dev" to > tell setuptools that if you can't find a released version >2.0, then > an in-development version is acceptable. I get that, but the question is, should such a version (which could be completely different tomorrow) appear by default e.g. when doing dependency resolution, where no ==dev is likely to be specified? > If they're using a "dev" version, then such a link is automatically > lower-precedence than anything else already, due to it being the > lowest available version.(Newer tools could treat it as 0dev or > whatever the official translation/suggestion is.) In addition, it > denotes a "non-stable" version, so if the tool allows one to > prioritize stable versions, it'll be eliminated as a candidate anyway > in that case. In the failure cases I was noting earlier, there were no other versions. I'm looking at the locate() mechanism not just as invoked directly from a user request to install (where ==dev might well be specified), but also via attempting to resolve dependencies - when it's not clear whether a version such as "dev" should really be included. > If the version tag is precise, OTOH, (i.e., something other than > 'dev'), then presumably the provider of the link can be trusted to > have identified what version it is. IIUC, those source control sites > let you download tarballs of arbitrary versions, so one could in > principle issue download releases of exact source snapshots. (Indeed, > it's not a bad way to go about it.) In cases where the URL specifies an archive named in the conventional way (name-version.ext), then these are currently picked up, even from DVCS hosting sites. What are not picked up are links which may contain the version number, but not necessarily using strictly defined conventions. In such cases, I would prefer not to add too many regex-style checks for various schemes which might be in use, but just provide a minimal way for users who need it to slot in these schemes via subclassing. After all, with the current setup, distlib is picking the same download URL as pip for 98% of cases. N.B. with the #egg=name-version scheme, as the fragment portion is not sent to the server, there's no way to expect that the server will always return a specific version (as no version is specified in the URL which goes to the server). At least that's not the case when you ask for a resource with a specific project name and version in the name of the resource itself. Regards, Vinay Sajip From pje at telecommunity.com Sun Oct 28 17:46:01 2012 From: pje at telecommunity.com (PJ Eby) Date: Sun, 28 Oct 2012 12:46:01 -0400 Subject: [Distutils] distlib locator compares reasonably well with pip's PackageFinder In-Reply-To: References: Message-ID: On Sat, Oct 27, 2012 at 7:08 PM, Vinay Sajip wrote: > N.B. with the #egg=name-version scheme, as the fragment portion is not sent > to the server, there's no way to expect that the server will always return > a specific version (as no version is specified in the URL which goes to the > server). That's just my point: version control sites (even ViewSVN) let you specify the version in the URL, just not in the filename portion. So it's quite valid to create a link to download a specifically designated version by specifying a revision control version, and then indicate the project name and version in the fragment identifier. IOW, the whole point of the fragment tag is to let you label a link that you know is to a specific version of a package, despite the filename portion not being explicit or unambiguous as to project name and version. > What are not picked up are links which may contain the version > number, but not necessarily using strictly defined conventions. In such cases, > I would prefer not to add too many regex-style checks for various schemes which > might be in use, but just provide a minimal way for users who need it to slot > in these schemes via subclassing. What I'm pointing out is that the existing #egg system provides a way to make links' project/version info unambiguous, so that you don't *need* lots of regex-based schemes, and people can just explicitly label their packages, even if due to hosting requirements they can't actually name their distributions according to the official scheme. (And this means that people won't need to be constantly writing plugins and subclasses to handle ever-more obscure hosting conventions.) In the future, of course, it might make sense to replace #egg with something like #pydist as the blessed syntax, while still processing #egg for backward-compatibility. From dholth at gmail.com Sun Oct 28 18:24:06 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 28 Oct 2012 13:24:06 -0400 Subject: [Distutils] distlib locator compares reasonably well with pip's PackageFinder In-Reply-To: References: Message-ID: Just call it a dist. Try not to overload the name quite as much as egg. On Oct 28, 2012 12:46 PM, "PJ Eby" wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Oct 28 19:47:43 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 28 Oct 2012 14:47:43 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: I think Metadata 1.3 is done. Who would like to czar? On Oct 22, 2012 12:53 PM, "Daniel Holth" wrote: > http://hg.python.org/peps/rev/50e8ea1a17a0 > > Based on the other "required" field's absence in the wild, only > "Metadata-Version", "Name", "Version", and "Summary" are required. > Hopefully a clearer explanation of 0, 1, or many occurrences of each > field. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Sun Oct 28 22:48:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Sun, 28 Oct 2012 17:48:31 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <20121018211014.73d56057@pitrou.net> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Now with an implementation at the end and possibly better wording for the required fields. I think it's feature complete. PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. The fields are "Setup-Requires-Dist" "Provides-Extra", and "Extension". This version also adds the `extra` variable to the `environment markers` specification and allows the description to be placed into a payload section. Metadata Files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with no maximum line length, followed by a blank line and an arbitrary payload. It is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Fields ====== This section specifies the names and semantics of each of the supported metadata fields. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the file. Metadata-Version :::::::::::::::: Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name :::: The name of the distribution. Example:: Name: BeagleVote Version ::::::: A string containing the distribution's version number. This field must be in the format specified in PEP 386. Example:: Version: 1.0a2 Summary ::::::: A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Platform (multiple use) ::::::::::::::::::::::: A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See "Classifier" below. Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) ::::::::::::::::::::::::::::::::: Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Description (optional, deprecated) :::::::::::::::::::::::::::::::::: A longer description of the distribution that can run to several paragraphs. Software that deals with metadata should not assume any maximum size for this field. The contents of this field can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs can also display the contents of the field as-is. This means that authors should be conservative in the markup they use. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in the description must be suffixed by whitespace to indicate continuation. Since Metadata 1.3 the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. Keywords (optional) ::::::::::::::::::: A list of additional keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) :::::::::::::::::::: A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ::::::::::::::::::::::: A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Author (optional) ::::::::::::::::: A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA Author-email (optional) ::::::::::::::::::::::: A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" Maintainer (optional) ::::::::::::::::::::: A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA Maintainer-email (optional) ::::::::::::::::::::::::::: A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" License (optional) :::::::::::::::::: Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ::::::::::::::::::::::::: Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Requires-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distutils project name (e.g., as found in the ``Name:`` field. optionally followed by a version declaration within parentheses. The distutils project names should correspond to names as found on the `Python Package Index`_. Version declarations must follow the rules described in `Version Specifiers`_ Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Setup-Requires-Dist (multiple use) :::::::::::::::::::::::::::::::::: Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time. Provides-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming a Distutils project which is contained within this distribution. This field *must* include the project identified in the ``Name`` field, followed by the version : Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For instance, source distributions of the ``ZODB`` project have historically included the ``transaction`` project, which is now available as a separate distribution. Installing such a source distribution satisfies requirements for both ``ZODB`` and ``transaction``. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ORM-bindings``, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Obsoletes-Dist (multiple use) ::::::::::::::::::::::::::::: Each entry contains a string describing a distutils project's distribution which this distribution renders obsolete, meaning that the two projects should not be installed at the same time. Version declarations can be supplied. Version numbers must be in the format specified in `Version Specifiers`_. The most common use of this field will be in case a project name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon distribution should be removed. Examples:: Obsoletes-Dist: Gorgon Obsoletes-Dist: OtherProject (<3.0) Requires-Python (optional) :::::::::::::::::::::::::: This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version numbers must be in the format specified in `Version Specifiers`_. Examples:: Requires-Python: 2.5 Requires-Python: >2.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 Requires-External (multiple use) :::::::::::::::::::::::::::::::: Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format specified in PEP 386: they should correspond to the version scheme used by the external dependency. Notice that there's is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a browsable URL for the project and a label for it, separated by a comma. Example:: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text limited to 32 signs. Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' A second distribution requires an optional dependency by placing it inside square brackets and can request multiple features by separating them with a comma (,). The full set of requirements is the union of the `Requires-Dist` sets evaluated with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose Two feature names `test` and `doc` are reserved to mark dependencies that are needed for running automated tests and generating documentation, respectively. It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. Extension (multiple use) :::::::::::::::::::::::: An ASCII string, not containing whitespace or the / character, that indicates the presence of extended metadata. Additional tags defined by an `Extension: Chili` must be of the form `Chili/Name`:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild An implementation might iterate over all the declared `Extension:` fields to invoke the processors for those extensions. As the order of the fields is not used, the `Extension: Chili` field may appear before or after its declared tags `Chili/Type:` etc. Version Specifiers ================== Version specifiers are a series of conditional operators and version numbers, separated by commas. Conditional operators must be one of "<", ">", "<=", ">=", "==" and "!=". Any number of conditional operators can be specified, e.g. the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. The comma (",") is equivalent to the **and** operator. Each version number must be in the format specified in PEP 386. When a version is provided, it always includes all versions that starts with the same value. For example the "2.5" version of Python will include versions like "2.5.2" or "2.5.3". Pre and post releases in that case are excluded. So in our example, versions like "2.5a1" are not included when "2.5" is used. If the first version of the range is required, it has to be explicitly given. In our example, it will be "2.5.0". Notice that some projects might omit the ".0" prefix for the first release of the "2.5.x" series: - 2.5 - 2.5.1 - 2.5.2 - etc. In that case, "2.5.0" will have to be explicitly used to avoid any confusion between the "2.5" notation that represents the full range. It is a recommended practice to use schemes of the same length for a series to completely avoid this problem. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding post or pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding post or pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, no matter wich one, excluding post or pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases of 2.6, pre and post releases of 2.7. It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would have include all versions of the 2.6.2 series. - ``Requires-Python: 2.5.0``: Equivalent to ">=2.5.0,<2.5.1". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding post or pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parenthesis are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in]?EXPR [or|and] ... where ``EXPR`` belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - Requires-Python - Requires-External - Requires-Dist - Setup-Requires-Dist - Provides-Dist - Obsoletes-Dist - Classifier (The `extra` variable is only meaningful for Requires-Dist.) Summary of Differences From PEP 345 =================================== * Metadata-Version is now 1.3. * Values are now expected to be UTF-8. * A payload (containing the description) may appear after the headers. * Added `extra` to environment markers. * Most fields are now optional. * Changed fields: - Requires-Dist - Description * Added fields: - Setup-Requires-Dist - Provides-Extra - Extension References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From p.f.moore at gmail.com Mon Oct 29 15:39:39 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 29 Oct 2012 14:39:39 +0000 Subject: [Distutils] distlib - experience using the library Message-ID: In order to get a feel for distlib, I spent a short time today writing a quick script to solve a problem I currently have. The results will hopefully be of some use in pointing out some "real world" experience with the library. The problem is as follows: I maintain a small library of sdists downloaded from PyPI, for offline use and to speed things up when I'm building virtualenvs. A maintenance issue with this approach is keeping the library reasonably up to date. It's not a huge issue, so I've never invested much time in writing anything to solve it, but it seemed a reasonable task for distlib. First of all, the positives: 1. It really was "a short time" I spent. Less time than it took me to write this mail. The locator API is beautifully easy to use, and fits this use case perfectly. 2. Although the resulting code is a quick hack, it's clean and readable. 3. Extending the code looks very straightforward. I haven't tried yet, but adding extra indexes, adding a download capability, all look simple. Areas where I found things lacking: I'd really like to have a means of getting all distributions from a locator. Something like the distlib.locators.get_all_distribution_names function, but on a locator. It would only be available on certain locators (specifically, the XMLRPC and directory ones) but for my script, it would have saved writing a directory scanning function and duplicating the filename parsing from the locator library. The dance to create a Distribution object from some data md = Metadata() md['Name'] = p md['Version'] = v dist = Distribution(md) cries out for a convenience method. On the other hand, I'm not sure how much advantage there is in using Distribution objects over just passing round the raw name and version strings for code as simple as mine. The version API seems very complex. I was expecting to just do something like Version(string) to get a sortable version object. Given that some of my packages use deliberately non-standard custom versions (2.3.x20121023) that's probably insufficient, but for this application I'm not really interested in validating versions. Using distutils.version.LooseVersion actually works better for me because it "just works". Is there some way to get the forgiving convenience of if distutils.version.LooseVersion(v1) > distutils.version.LooseVersion(v2): # whatever with distlib? But regardless, these are minor quibbles. The overall experience was very positive. Paul. From vinay_sajip at yahoo.co.uk Mon Oct 29 16:11:55 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 29 Oct 2012 15:11:55 +0000 (UTC) Subject: [Distutils] distlib - experience using the library References: Message-ID: Paul Moore gmail.com> writes: > In order to get a feel for distlib, I spent a short time today writing > a quick script to solve a problem I currently have. The results will > hopefully be of some use in pointing out some "real world" experience > with the library. Thanks very much for the feedback. > I'd really like to have a means of getting all distributions from a > locator. Something like the > distlib.locators.get_all_distribution_names function, but on a Okay, I'll look at that. > The dance to create a Distribution object from some data > md = Metadata() > md['Name'] = p > md['Version'] = v > dist = Distribution(md) > cries out for a convenience method. On the other hand, I'm not sure > how much advantage there is in using Distribution objects over just > passing round the raw name and version strings for code as simple as > mine. There are still plenty of rough edges in this area, but I hope to smooth them out in due course. > The version API seems very complex. I was expecting to just do > something like Version(string) to get a sortable version object. Given > that some of my packages use deliberately non-standard custom versions > (2.3.x20121023) that's probably insufficient, but for this application > I'm not really interested in validating versions. Using It's probably the docs that need sorting out. With versions, distutils2 has NormalizedVersion which conforms to PEP 386 (though there has been some discussion here on changing the ordering of 'dev' w.r.t. other qualifiers). However, any work with existing projects on PyPI means you have to consider supporting other versions schemes, so I've added support for: LegacyVersion - as per setuptools/distribute SemanticVersion - as per semver.org AdaptiveVersion - tries PEP-386, then the suggestion mechanism for PEP 386, and then semantic I'm not saying that all these version classes necessarily have to live on, but it's useful to have them around at this stage in distlib's development, if only for experimenting with. Any of the above will give you an orderable version (you can only compare instances of the same type, of course): >>> from distlib.version import LegacyVersion as Version >>> va, vb, vc, vd, vf, vp = [Version(s) for s in ('1.0-a1', '1.0-b2', '1.0rc3', '1.0-dev', '1.0', '1.0p1')] >>> va < vb True >>> va < vc True >>> va < vd False >>> va < vf True >>> va < vp True >>> vf < vp True >>> vc < vf True >>> > But regardless, these are minor quibbles. The overall experience was > very positive. Good to know. Thanks again for the feedback. Regards, Vinay Sajip From p.f.moore at gmail.com Mon Oct 29 16:50:23 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 29 Oct 2012 15:50:23 +0000 Subject: [Distutils] distlib - experience using the library In-Reply-To: References: Message-ID: On 29 October 2012 15:11, Vinay Sajip wrote: > Paul Moore gmail.com> writes: >> The version API seems very complex. I was expecting to just do >> something like Version(string) to get a sortable version object. Given >> that some of my packages use deliberately non-standard custom versions >> (2.3.x20121023) that's probably insufficient, but for this application >> I'm not really interested in validating versions. Using > > It's probably the docs that need sorting out. With versions, distutils2 has > NormalizedVersion which conforms to PEP 386 (though there has been some > discussion here on changing the ordering of 'dev' w.r.t. other qualifiers). Yes, I did think it was more likely a documentation question rather than a functionality question. > However, any work with existing projects on PyPI means you have to consider > supporting other versions schemes, so I've added support for: > > LegacyVersion - as per setuptools/distribute > SemanticVersion - as per semver.org > AdaptiveVersion - tries PEP-386, then the suggestion mechanism for PEP 386, > and then semantic LegacyVersion did the job fine for me. > I'm not saying that all these version classes necessarily have to live on, but > it's useful to have them around at this stage in distlib's development, if > only for experimenting with. Agreed. From a user experience point of view, I tend to only care about the case where I'm dealing with versions created by someone else, and I really don't care whether those versions are "valid" in any purist sense, only that nothing I pass in will cause a parse error, and the resulting version objects order sanely. My perspective on "sanely" is limited pretty much to release versions, so "any series of numbers separated by dots orders as if it were a tuple of numbers, anything with non-numeric data in I don't really care about" covers it really. Version semantics is a whole other thread though - I won't say more on that topic here :-) Paul. From dholth at gmail.com Wed Oct 31 13:38:08 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 31 Oct 2012 08:38:08 -0400 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> References: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> Message-ID: On Wed, Oct 24, 2012 at 7:04 AM, Ronald Oussoren wrote: > > On 18 Oct, 2012, at 19:29, Daniel Holth wrote: > >> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >> been stable since May and we are preparing a patch to support it in >> pip, but we need to earn consensus before including it in the most >> widely used installer. > > PEP 425: > > * "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead" > > I don't particularly like replacing dots by underscores. That needed because you use the dot character in compressed tag sets, but why not use a comma to separate items in the compressed tag set? > > * "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." > > Why the replacement? The need for replacement could be avoided by using a different separator between elements of a tag (for example "~" or "+"), and furthermore the platform tag is at a know > location, and hence the use of hyphens in the platform tag is harmless (use "python_tag, abi_tag, platform_tag = tag.split('-', 2)" to split the tag into its elements. > > * "compressed tag sets" > > Using '"," instead of "." to separate elements of the tag set takes away the need to replace dots in tag elements, and seems more natural to me (you'd also use comma to separate the elements > when you write them down in prose or python code. I can't get excited about changing the convention. The hyphen-to-underscore folding and - separated file parts is the same as what setuptools uses. _ folding for . as well in the compat tags? It may not be beautiful but it is very unlikely to cause ambiguity. From ronaldoussoren at mac.com Wed Oct 31 14:29:37 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 31 Oct 2012 14:29:37 +0100 Subject: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: References: <8DC03FC0-5182-4979-B219-F1FA5E9EB56E@mac.com> Message-ID: <1486488E-6CA0-41CE-806C-EF5F0B71E833@mac.com> On 31 Oct, 2012, at 13:38, Daniel Holth wrote: > On Wed, Oct 24, 2012 at 7:04 AM, Ronald Oussoren wrote: >> >> On 18 Oct, 2012, at 19:29, Daniel Holth wrote: >> >>> I'd like to submit the Wheel PEPs 425 (filename metadata), 426 >>> (Metadata 1.3), and 427 (wheel itself) for acceptance. The format has >>> been stable since May and we are preparing a patch to support it in >>> pip, but we need to earn consensus before including it in the most >>> widely used installer. >> >> PEP 425: >> >> * "The version is py_version_nodot. CPython gets away with no dot, but if one is needed the underscore _ is used instead" >> >> I don't particularly like replacing dots by underscores. That needed because you use the dot character in compressed tag sets, but why not use a comma to separate items in the compressed tag set? >> >> * "The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _." >> >> Why the replacement? The need for replacement could be avoided by using a different separator between elements of a tag (for example "~" or "+"), and furthermore the platform tag is at a know >> location, and hence the use of hyphens in the platform tag is harmless (use "python_tag, abi_tag, platform_tag = tag.split('-', 2)" to split the tag into its elements. >> >> * "compressed tag sets" >> >> Using '"," instead of "." to separate elements of the tag set takes away the need to replace dots in tag elements, and seems more natural to me (you'd also use comma to separate the elements >> when you write them down in prose or python code. > > I can't get excited about changing the convention. The > hyphen-to-underscore folding and - separated file parts is the same as > what setuptools uses. _ folding for . as well in the compat tags? It > may not be beautiful but it is very unlikely to cause ambiguity. Beauty is also important. BTW. Setuptools does not fold '-' to underscore in the tag: xs_image-1.0-py2.7-linux-x86_64.egg This would be "xs_image-1.0-cp27-cp27-linux_x86_64.whl" in your scheme (a binary package for CPython 2.7, platform is Linux x86_64). Replacing characters is especially ugly on OSX, compare "xs_image-1.0-cp27-cp27-macosx_10_3_fat.whl" and "xs_image-1.0-cp27-cp27-macosx-10.3-fat.whl". As I've written before, it is not necessary to replace characters in the platform tag because it is the last part of the filename anyway. Ronald