From robertc at robertcollins.net Wed Mar 2 19:43:33 2016 From: robertc at robertcollins.net (Robert Collins) Date: Thu, 3 Mar 2016 13:43:33 +1300 Subject: [Distutils] abstract build system approaches redux Message-ID: This is my attempt to consolidate the three different proposals we have at the moment into a single set of design choices. We have three draft designs - from Nathaniel (517), Donald (unnumberd) and mine (516). For purposes of comparison, I've skipped all rationale here, and only listed the highlights that differ across the proposals. If you haven't read all three, you should before continuing. My goal is to let us pick a final shape - and then we can shave the rough edges off of that shape to get a final PEP (or PEPs) 517: - Python interface to the system - build requires, optional wheel metadata (to directory on disk), build wheel. install editable commands 516: - CLI interface to the system - build requires, mandatory wheel metadata, build wheel, install editable commands Donald's: - Python interface to the system - defines new 'source wheel' concept - source_metadata, source_wheel, binary_metadata, binary_wheel commands - 'develop' // editable not addressed currently - outputs directories on disk, rather than zipped up things Ok, now here's my view on the differences... Python vs CLI - I don't care enough to argue. I think a CLI is better in this space, as running commands is the lingua franca of CLI's; previously Donald has advocated for that as well, but his proposal is now a Python API. *shrug*. I want the BDFL-Delegate to choose. 516 and 517 are otherwise identical in all meaningful ways - there's plenty of stuff to bikeshed on (e.g. should wheel metadata be optional or mandatory), but I don't think it is crucial: we can always change this in future by issuing a new version. Its not free, but if we can't agree, we can at least find out later. Donalds and 516 both have the ability to get metadata out explicitly as a mandatory thing, so I think we should make it mandatory and weaken it in future if e.g. flit eventually come back saying its a burden. [I don't believe it will be]. File formats - BDFL-Delegate can pick. I so don't care :). New source format. I'm quite strongly against this at the moment: Building from VCS is now a fairly major thing in the community, and we can't mandate new URLs for all sources. We're going to thus be forcing a change to all tooling (e.g. Debian and Fedora and pip and others) that know how to build things from an arbitrary VCS url. A new source extension in pypi doesn't reduce the impact of that at all, and the necessary code to handle that will also handle sdists's on pypi that have the new config file and may be missing a setup.py. If there is a new source extension in pypi, we should expect that folk using e.g. flit will *not* upload old style sdists to pypi - they don't at the moment, and the objection to doing so will *remain*. -> We don't save any grief or friction for the adoption of the new thing, nor enable new build systems any more easily. A shim to the new shiny would enable current sdists for anything adopting the new thing in exactly the same manner as it would if we reuse the extension. I can see an argument for a new iteration of the sdist format for better metadata, and to be able to tell what metadata is trustable etc, but I don't see that as related to the problem of enabling third party build systems - and since all the easy cases will be solved by binary wheels I think we're in rapidly diminishing territory here - if something can't upload a binary wheel, its quite likely to have complex dependencies - and we haven't solved the numpy ABI problem yet, so I'd like to actually focus on that well and solve it - and once solved see if we can retrofit it into the existing sdist format, or if a new one is actually needed. Develop is essential to tackle IMO. We can either fully PEP define it now - and if so we should stop talking about this and start talking about that. If we don't have bandwidth to do that yet (and AFAICT we don't, based on IRC discussions and the headaches of interactions with the three different namespace package possibilities, virtualenvs and user installs....). Outputting the wheel as an unzipped directly on disk would be a nice optimisation for big trees, I think, so perhaps we should apply that to 516/517 to reduce the set of things we're discussing. -Rob 516: https://github.com/pypa/interoperability-peps/blob/master/pep-0516-build-system-abstraction.rst 517: https://github.com/pypa/interoperability-peps/blob/master/pep-0517-build-system-abstraction.rst Donalds: https://github.com/pypa/interoperability-peps/compare/master...dstufft:build-pipeline -- Robert Collins Distinguished Technologist HP Converged Cloud From donald at stufft.io Wed Mar 2 20:22:36 2016 From: donald at stufft.io (Donald Stufft) Date: Wed, 2 Mar 2016 20:22:36 -0500 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: I'm hoping to get some time to finish fleshing mine out this weekend. I've been unable to do much except in small spurts this last week due to some complications from a temporary filling after a root canal last week (it threw my bite off and I've been having jaw pain + feeling of drunkiness + extreme exhaustion). The temporary filling is coming out tomorrow so hopefully that will be cleared up by then. Sent from my iPhone > On Mar 2, 2016, at 7:43 PM, Robert Collins wrote: > > We have three draft designs - from Nathaniel (517), Donald (unnumberd) > and mine (516). From ncoghlan at gmail.com Thu Mar 3 08:19:44 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 3 Mar 2016 23:19:44 +1000 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: On 3 March 2016 at 10:43, Robert Collins wrote: > Python vs CLI - I don't care enough to argue. I think a CLI is better > in this space, as running commands is the lingua franca of CLI's; > previously Donald has advocated for that as well, but his proposal is > now a Python API. *shrug*. I want the BDFL-Delegate to choose. I'm happy to let the discussion run for the other open questions you mention, but I'm now prepared to pronounce on this aspect based on one specific factor: I don't want to mandate that all future build systems for Python projects (whether front ends or back ends) must be written in Python. If somebody comes up with an all-singing, all-dancing Python build system that happens to be written in Go, or Rust, or Haskell, or that they made by adapting an existing build system for another language ecosystem to have native support for also building Python packages, I'd like for the Python packaging ecosystem to be able to handle that without significant fuss. Attaining that level of cross-language interoperability then necessarily means defining the formal build system abstraction interface at the CLI boundary, rather than as a Python API. Defining a helper library/framework to make it *easier* to write new build systems in Python would still make sense, but that can be done in the usual way that any other library gains widespread acceptance: by being easier to use than a "DIY" approach when it comes to directly implementing the underlying interoperability specification. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Thu Mar 3 08:44:56 2016 From: donald at stufft.io (Donald Stufft) Date: Thu, 3 Mar 2016 08:44:56 -0500 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: > On Mar 3, 2016, at 8:19 AM, Nick Coghlan wrote: > > On 3 March 2016 at 10:43, Robert Collins wrote: >> Python vs CLI - I don't care enough to argue. I think a CLI is better >> in this space, as running commands is the lingua franca of CLI's; >> previously Donald has advocated for that as well, but his proposal is >> now a Python API. *shrug*. I want the BDFL-Delegate to choose. > > I'm happy to let the discussion run for the other open questions you > mention, but I'm now prepared to pronounce on this aspect based on one > specific factor: I don't want to mandate that all future build systems > for Python projects (whether front ends or back ends) must be written > in Python. > > If somebody comes up with an all-singing, all-dancing Python build > system that happens to be written in Go, or Rust, or Haskell, or that > they made by adapting an existing build system for another language > ecosystem to have native support for also building Python packages, > I'd like for the Python packaging ecosystem to be able to handle that > without significant fuss. > > Attaining that level of cross-language interoperability then > necessarily means defining the formal build system abstraction > interface at the CLI boundary, rather than as a Python API. > > Defining a helper library/framework to make it *easier* to write new > build systems in Python would still make sense, but that can be done > in the usual way that any other library gains widespread acceptance: > by being easier to use than a "DIY" approach when it comes to directly > implementing the underlying interoperability specification. I'd like to push back against this, speaking as someone who was originally pro CLI: I think that a Python API is actually better for one reason: introspection. I cannot think of a particularly great way to have a CLI based build tool *evolve* with new APIs that are not user facing without requiring end users to do something like mark "ok now my thing is X compatible" or without inventing some sort of protocol negotiation phase. For instance, let's say that we say that we need an ``metadata`` API that can output metadata 1.x style metadata. That's great and right now we can just assume that it exists. However, let's say in the future we add a metadata 2.0 spec that isn't backwards compatible. Now how do we handle this? A new version of pip can't just assume that the ``metadata`` command is going to always return 2.x metadata now, because that would break all existing build tools so it needs some method to determine what capabilities a particular build system has. I see a few options: * If we're using a CLI based thing, we need to create a negotiation phase, which I think is a bad idea because it's kind of complicated and error prone. Looking at TLS or HTTP, there's more than one bug that exists because of this. * If we're using a CLI based thing, we need to bake into the file what the capabilities of the build system is. Maybe using something like: generate-metadata-2: true # OR metadata2-generation-command: ${PYTHON} -m generate-metdata However, I think this is bad because it makes it harder to get automatic "wins" for new features because you have to convince every single author to go and modify their file to tell pip (or whatever) that this new feature is available as well as then needing to force a >= in their build dependencies because they mandating some new feature by baking it directly into their project. * If we're using a Python based API, we can simply just say that there can be no backwards incompatible changes made to an API name once defined. If we want to create a new function that produces the new style metadata, then we can simply do a check like: if hasattr(api, "metadata2"): # Do Metadata 2.x Code else: # Do Metadata 1.x Code This not only allows all projects to automatically start using this new feature as soon as a build tool implements it, getting automatic wins, but it also makes it trivial for a project to support both old and new versions of pip, since there will be a different API name for a backwards incompatible change. Old pip would just use the old code, and new pip would use the new code. The only real benefits I can see to using a CLI based over a Python based are: * We move some code out of pip into each individual build backend, making it clearer the process boundaries. This is a nice thing, and it is the primary reason I wanted a CLI based API. However I don't think it outweighs the ability to introspect what API methods are available. * We make it easier to use a non Python based build system. This is also a nice thing, however I don't think it should be a major decider in what API we provide. Any reasonable build system is going to have to be available via ``pip install`` in some fashion, so even if you write your build system in Go or Rust, you're going to have to create a Python package for it anyways, and if you're doing that, adding a tiny shim is pretty trivial, something like: import os.path import subprocess BIN = os.path.join(os.path.dirname(__file__), "mytool") def some_api_function(*args, **kwargs): flags = convert_args_kwargs_to_flags(*args, **kwargs) subprocess.run(BIN, *flags, check=True) I don't believe it to be a substantial burden to need to write a tiny wrapper if you're going to do something which I believe is going to be very unlikely. In the end, I think this comes down that we shouldn't optimize for the least common case, at the expense of the ability to more easily evolve the API in the future. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From solipsis at pitrou.net Thu Mar 3 08:57:16 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 3 Mar 2016 14:57:16 +0100 Subject: [Distutils] abstract build system approaches redux References: Message-ID: <20160303145716.568c5722@fsol> On Thu, 3 Mar 2016 08:44:56 -0500 Donald Stufft wrote: > > I'd like to push back against this, speaking as someone who was originally pro > CLI: > > I think that a Python API is actually better for one reason: introspection. > I cannot think of a particularly great way to have a CLI based build tool > *evolve* with new APIs that are not user facing without requiring end users > to do something like mark "ok now my thing is X compatible" or without > inventing some sort of protocol negotiation phase. I'll add that some build systems may have a non-trivial startup cost (for example conda-build sets up an isolated environment with well-defined binaries in it), therefore issuing several CLI commands can be significantly more costly (and/or difficult to optimize for) than issuing several API calls from the same single process invocation. Regards Antoine. From ncoghlan at gmail.com Thu Mar 3 09:11:39 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 4 Mar 2016 00:11:39 +1000 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: On 3 March 2016 at 23:44, Donald Stufft wrote: > * We make it easier to use a non Python based build system. This is also a nice > thing, however I don't think it should be a major decider in what API we > provide. Any reasonable build system is going to have to be available via > ``pip install`` in some fashion, so even if you write your build system in > Go or Rust, you're going to have to create a Python package for it anyways, > and if you're doing that, adding a tiny shim is pretty trivial, something > like: > > import os.path > import subprocess > > BIN = os.path.join(os.path.dirname(__file__), "mytool") > > def some_api_function(*args, **kwargs): > flags = convert_args_kwargs_to_flags(*args, **kwargs) > subprocess.run(BIN, *flags, check=True) > > I don't believe it to be a substantial burden to need to write a tiny wrapper > if you're going to do something which I believe is going to be very unlikely. Ah, you're right, I hadn't accounted for the fact the same shim that makes a non-Python tool installable as a build dependency could also handle the adaptation from a Python API to a CLI or FFI based approach, so putting the standardised interface in Python doesn't raise any insurmountable barriers to cross-language interoperability - they just move the additional complexity to the less common case. Given that, I'm going to go back to reserving judgement on *all* of the points Robert mentioned, at least until you've had a chance to finish writing up your own proposal - the determining factor I thought I had found turned out not to be so determining after all :) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Mar 3 11:51:28 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 3 Mar 2016 16:51:28 +0000 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: On 3 March 2016 at 13:44, Donald Stufft wrote: > * If we're using a CLI based thing, we need to create a negotiation phase, > which I think is a bad idea because it's kind of complicated and error prone. > Looking at TLS or HTTP, there's more than one bug that exists because of > this. Additionally, it seems to me that a negotiation means additional round-trips to the build system. Which means extra subprocess calls. On Windows in particular, the cost of a subprocess call is not that cheap, so this could result in increased build times. I understand that the actual build steps will likely always involve a subprocess, for isolation purposes. But avoiding subprocess calls for "admin" issues like "what version of the protocol do you support?" is a worthwhile goal. (Beyond noting the above as a data point, I don't really have a strong opinion on a CLI vs Python interface, though). Paul From ich at ronnypfannschmidt.de Thu Mar 3 14:30:22 2016 From: ich at ronnypfannschmidt.de (Ronny Pfannschmidt) Date: Thu, 3 Mar 2016 20:30:22 +0100 Subject: [Distutils] Idea: Positive/negative extras in general and as replacement for features In-Reply-To: References: <567061F9.1040500@ronnypfannschmidt.de> Message-ID: <56D890CE.1080101@ronnypfannschmidt.de> Am 09.02.2016 um 22:24 schrieb Robert Collins: > Sorry for not replying for so long. > > On 16 December 2015 at 07:54, Ronny Pfannschmidt > wrote: >> Hello everyone, >> >> the talk about the sqlalchemy feature extra got me thinking >> >> what if i could specify extras that where installed by default, but >> users could opt out >> >> a concrete example i have in mind is the default set of pytest plugins >> i would like to be able to externalize legacy support details of pytest >> without needing people to suddenly depend on pytest[recommended] instead >> of pytest to have what they know function as is >> >> instead i would like to be able to do something like a dependency on >> pytest[-nose,-unittest,-cache] to subtract the items > > So the challenge here will be defining good, useful, and predictable > behaviour when the dependency graph is non-trivial. > > Using your example, what should pip do when told > > pip install pytest[nose, -unittest] proj2 > > and proj2 depends on pytest[unittest] > > ? those should be additive, negative extras should be used purely to leave out dependencies for a single requirement if something else needs them, then they should of course be included even a dependency on pytest itself should include the unittest dependency negative requirements should be absolutely local and only effect the dependency graph of a single direct dependency so if there is another dependency without that negation, then it should consider the full graph in any case > If it installs pytest[unittest], then the first pytest dependency was > no honoured. If it does not install pytest[unittest], then the proj2 > dependencies were not honoured. So it must error. > > -> Which means that using a [-THING] clause anywhere is going to be > super fragile, as indeed 'never install with X' things are in > distributions - its much better to find ways to express things purely > additively IMO. > > There are many more complex interactions possible with the + / - DSL > you've sketched - and I don't think they are reducible - that is, its > not the DSL per se, but the basic feature of allowing cuts in the > graph traversal that lead to that complexity. If we can come up with > good, consistent, predictable answers, even considering three-party > interactions, then I've no objection per se: but I think that will be > very hard to do. I certainly don't have time at the moment to help - > sorry :(. > > -Rob > > > > From tleeuwenburg at gmail.com Tue Mar 8 23:40:22 2016 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Wed, 9 Mar 2016 15:40:22 +1100 Subject: [Distutils] Avoiding cursed versions... Message-ID: I'm not aware of a way to avoid 'cursed versions' of packages. For example, pandas 0.17.0 has an issue with duplicate rows. This bug is fixed in 0.17.1, but also the previous version was fine. I apologies if there is logic to handle this situation. I would like to express this as a "latest version but not 0.17.0". This is because 'latest' might mean different things depending where you are getting your versions from (e.g. anaconda, pypi, OS packages). I don't know what to suggest, but I thought this was an interesting use case and worth mentioning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jp at jamezpolley.com Tue Mar 8 23:59:29 2016 From: jp at jamezpolley.com (James Polley) Date: Wed, 9 Mar 2016 15:59:29 +1100 Subject: [Distutils] Avoiding cursed versions... In-Reply-To: References: Message-ID: pandas!=0.17.0,>=0.16.0 should do what you need, I think On Wed, Mar 9, 2016 at 3:40 PM, Tennessee Leeuwenburg < tleeuwenburg at gmail.com> wrote: > I'm not aware of a way to avoid 'cursed versions' of packages. For > example, pandas 0.17.0 has an issue with duplicate rows. This bug is fixed > in 0.17.1, but also the previous version was fine. I apologies if there is > logic to handle this situation. > > I would like to express this as a "latest version but not 0.17.0". This is > because 'latest' might mean different things depending where you are > getting your versions from (e.g. anaconda, pypi, OS packages). > > I don't know what to suggest, but I thought this was an interesting use > case and worth mentioning. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tleeuwenburg at gmail.com Wed Mar 9 00:02:43 2016 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Wed, 9 Mar 2016 16:02:43 +1100 Subject: [Distutils] Avoiding cursed versions... In-Reply-To: References: Message-ID: Thanks James, sorry for the n00b level question. On 9 March 2016 at 15:59, James Polley wrote: > pandas!=0.17.0,>=0.16.0 > > > should do what you need, I think > > > On Wed, Mar 9, 2016 at 3:40 PM, Tennessee Leeuwenburg < > tleeuwenburg at gmail.com> wrote: > >> I'm not aware of a way to avoid 'cursed versions' of packages. For >> example, pandas 0.17.0 has an issue with duplicate rows. This bug is fixed >> in 0.17.1, but also the previous version was fine. I apologies if there is >> logic to handle this situation. >> >> I would like to express this as a "latest version but not 0.17.0". This >> is because 'latest' might mean different things depending where you are >> getting your versions from (e.g. anaconda, pypi, OS packages). >> >> I don't know what to suggest, but I thought this was an interesting use >> case and worth mentioning. >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Mar 9 04:06:29 2016 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 9 Mar 2016 10:06:29 +0100 Subject: [Distutils] Idea: Positive/negative extras in general and as replacement for features In-Reply-To: <56D890CE.1080101@ronnypfannschmidt.de> References: <567061F9.1040500@ronnypfannschmidt.de> <56D890CE.1080101@ronnypfannschmidt.de> Message-ID: On Thu, Mar 3, 2016 at 8:30 PM, Ronny Pfannschmidt wrote: > > > Am 09.02.2016 um 22:24 schrieb Robert Collins: >> >> Sorry for not replying for so long. >> >> On 16 December 2015 at 07:54, Ronny Pfannschmidt >> wrote: >>> >>> Hello everyone, >>> >>> the talk about the sqlalchemy feature extra got me thinking >>> >>> what if i could specify extras that where installed by default, but >>> users could opt out >>> >>> a concrete example i have in mind is the default set of pytest plugins >>> i would like to be able to externalize legacy support details of pytest >>> without needing people to suddenly depend on pytest[recommended] instead >>> of pytest to have what they know function as is >>> >>> instead i would like to be able to do something like a dependency on >>> pytest[-nose,-unittest,-cache] to subtract the items >> >> >> So the challenge here will be defining good, useful, and predictable >> behaviour when the dependency graph is non-trivial. >> >> Using your example, what should pip do when told >> >> pip install pytest[nose, -unittest] proj2 >> >> and proj2 depends on pytest[unittest] >> >> ? > > those should be additive, > negative extras should be used purely to leave out dependencies for a single > requirement > if something else needs them, then they should of course be included > > even a dependency on pytest itself should include the unittest dependency > > negative requirements should be absolutely local and only effect the > dependency graph of a single direct dependency > > so if there is another dependency without that negation, then it should > consider the full graph in any case I only just saw this thread because Ronny's recent(-ish) reply, and I agree with him. In my mind the use case for something like this is pretty independent of "more complex" interactions like Robert's example. I think the idea here is that if I run `pip install pytest[-unittest]` I'm not saying the `pytest[-unittest]` must never ever be installed on the system. Rather, it's useful in a far more limited cases where I need more fine-grained control over what's installed--think for example application bundles or running on embedded systems. A case that would be useful to me would be to have an `astropy[tests]` extra that is installed by default. The astropy test suite is a bit weighty, especially due to the size of test files. I've proposed making the tests an extra before but that was shot down (I think rightfully) since it would be preferable to have it installed by default. However, for cases where the tests take up too much space a `pip install astropy[-tests]` would be nice--there's not likely to be an additional package that depends on `astropy[tests]` and even if there were I'd know in building my system whether or not I really need it. The alternative is post-install stripping down of packages, which can be unpredictable and error-prone in my experience. Erik From cournape at gmail.com Fri Mar 11 04:41:19 2016 From: cournape at gmail.com (David Cournapeau) Date: Fri, 11 Mar 2016 09:41:19 +0000 Subject: [Distutils] PEP 425 vs pip Message-ID: Hi, At Enthought we have been trying to use PEP 425 tags to include its metadata in our packages. It has been harder than expected because of missing information in PEP 425, and maybe inconsistencies with what pip actually does. I think PEP 425 should be updated to take into account pip actual implementation: 1. python tag and pypy: the PEP says "the version is py_version_nodot . CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18 , pp19 ". This is not what pip does as of version 8.1.0 for pypy. Instead, the version is python major + sys.pypy_version_info.major + sys.pypy_version_info.minor. 2. It would be nice to know how to deal with other alternative implementations as well. Would the scheme in 1. work for jython, etc... ? 3. The abi tag should at least be specified for cpython IMO. E.g. the meaning of `m`, `u` and `d` should be clearly specified. While the PEP mentions each implementation community is welcome to use the abi tag as they wish, given that in practice it is defined in pip, I think that should be specified as well. Do the above make sense ? If so, I could spend some time to update it (or create a new amending PEP ?) thanks, David -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 11 10:26:55 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 12 Mar 2016 01:26:55 +1000 Subject: [Distutils] PEP 425 vs pip In-Reply-To: References: Message-ID: On 11 March 2016 at 19:41, David Cournapeau wrote: > Hi, > > At Enthought we have been trying to use PEP 425 tags to include its metadata > in our packages. It has been harder than expected because of missing > information in PEP 425, and maybe inconsistencies with what pip actually > does. > > I think PEP 425 should be updated to take into account pip actual > implementation: > > 1. python tag and pypy: the PEP says "the version is py_version_nodot . > CPython gets away with no dot, but if one is needed the underscore _ is used > instead. PyPy should probably use its own versions here pp18 , pp19 ". This > is not what pip does as of version 8.1.0 for pypy. Instead, the version is > python major + sys.pypy_version_info.major + sys.pypy_version_info.minor. > > 2. It would be nice to know how to deal with other alternative > implementations as well. Would the scheme in 1. work for jython, etc... ? > > 3. The abi tag should at least be specified for cpython IMO. E.g. the > meaning of `m`, `u` and `d` should be clearly specified. > > While the PEP mentions each implementation community is welcome to use the > abi tag as they wish, given that in practice it is defined in pip, I think > that should be specified as well. > > Do the above make sense ? If so, I could spend some time to update it (or > create a new amending PEP ?) Proposed errrata & minor admentments for PEP 425 can be submitted as a PyPUG issue with a PR against https://packaging.python.org/en/latest/specifications/#platform-compatibility-tags (the actual technical details of the proposed changes should still be discussed here) This does remind me I have unfinished work in officially amending that spec management process, though: https://github.com/pypa/pypa.io/issues/11#issuecomment-195412332 Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tritium-list at sdamon.com Sat Mar 12 05:44:35 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 12 Mar 2016 05:44:35 -0500 Subject: [Distutils] [Python-ideas] Version control system link in package meta data In-Reply-To: References: Message-ID: <56E3F313.7060908@sdamon.com> (cross-replying to distutils-sig at python.org) For a tool like this to work reliably, the semantics of the project urls would have to change (as of right now, there is no set requirement that a VCS link in project URL's point to ), nor is there a requirement that the field even be present. For this to be useful for your tool, projects would have to be required, or at least heavily encouraged, to put VCS links in their pypi listings. Moreover, such a requirement would additionally need to limit what type of VCS links you post as to make the tool reasonable to maintain. (Not everyone uses git, let alone github. Do you want to write a tool that supports Team Foundation Server?) Another option would be to pull in source packages with your tool, but that too would require that people actually post source packages for their projects on PyPI. There is no such requirement as it stands, and the number of pure-python projects posting sdists may actually decrease with the uptake of universal wheels. Automating this process is not impossible. As you said, you could write your own tool to do it, but as it stands, even with that tool, you would have to do quite a bit of manual legwork still. I agree, it would be nice if everyone used git (or any of a small set of VCS), and all the packages on pypi listed their repositories in the metadata. If that were the case, this tool might already exist. In the current state of things, though, i don't think it makes much sense to produce a general purpose tool for this. On 3/12/2016 05:12, Tymoteusz Jankowski wrote: > Hi, > tldr: install project and its requirements as cloned repositories (if > possible) > > Let's say I'm developing requests > library which relies on > these > packages. > My workflow is this: > > $ git clone https://github.com/kennethreitz/requests.git > $ cd requests > $ virtualenv requests > $ . requests/bin/activate.fish > $ pip install -r requirements.txt > > Now I can change Requests library easily and commit changes, but when > i have to change a library from Requests' requirements I have to clone > and reinstall library (It's boring + I'm lazy). > I need tool that works like this: > > $ magic-command install requests > 1 the tool checks in Requests package meta where sources are stored > 2 clone the source > 3 do the same for each package from requirements (or fallback to > current method) > > I could write the tool and be the only one in the world using it ;) > but there should be an option for storing repository link. > Could you advice anything? > I found this: > https://www.python.org/dev/peps/pep-0345/#project-url-multiple-use > but > 1 I'm not sure it's right option > 2 I can't see handler for this option in distutils > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 12 09:46:30 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 13 Mar 2016 00:46:30 +1000 Subject: [Distutils] [Python-ideas] Version control system link in package meta data In-Reply-To: <56E3F313.7060908@sdamon.com> References: <56E3F313.7060908@sdamon.com> Message-ID: On 12 March 2016 at 20:44, Alexander Walters wrote: > I agree, it would be nice if everyone used git (or any of a small set of > VCS), and all the packages on pypi listed their repositories in the > metadata. If that were the case, this tool might already exist. In the > current state of things, though, i don't think it makes much sense to > produce a general purpose tool for this. We don't place any particular requirements on the development practices of projects publishing their releases through PyPI, so there's no requirement for a public VCS URL to even exist for a project, let alone for it to be mentioned in the project metadata. That said, since project URLs do make it possible for projects to share that metadata if they want to, this is a situation where a "checkout-pypi-project" that gained popularity might provide more incentive for maintainers to provide that metadata and keep it up to date. As a fallback for projects without that metadata, searching popular hosting sites like GitHub, BitBucket, GitLab and even SourceForge, would provide some initial links to investigate. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tritium-list at sdamon.com Sat Mar 12 16:08:32 2016 From: tritium-list at sdamon.com (Alexander Walters) Date: Sat, 12 Mar 2016 16:08:32 -0500 Subject: [Distutils] [Python-ideas] Version control system link in package meta data In-Reply-To: References: <56E3F313.7060908@sdamon.com> Message-ID: <56E48550.1020409@sdamon.com> I think that reiterates my point - the information infrastructure does not exist to make a tool that reliably works in the general case... today. If Tymoteusz Jankowski develops a tool for their workflow, and it works, and they release it, maybe the status quo will change. I think I am speaking to the reality of the situation in that there is a lot of momentum (dead weight not moving has momentum, too) in not being universally consistent with VCS urls on pypi. I suspect that there will have to be a buildup of momentum opposing that before pypa (or some other interested body) moves to make the tool suggested really happen. ... Which I guess means Tymoteusz Jankowski should probably get an alpha of such a tool out, and that might provoke the movement needed for the tool to succeed. On 3/12/2016 09:46, Nick Coghlan wrote: > On 12 March 2016 at 20:44, Alexander Walters wrote: >> I agree, it would be nice if everyone used git (or any of a small set of >> VCS), and all the packages on pypi listed their repositories in the >> metadata. If that were the case, this tool might already exist. In the >> current state of things, though, i don't think it makes much sense to >> produce a general purpose tool for this. > We don't place any particular requirements on the development > practices of projects publishing their releases through PyPI, so > there's no requirement for a public VCS URL to even exist for a > project, let alone for it to be mentioned in the project metadata. > > That said, since project URLs do make it possible for projects to > share that metadata if they want to, this is a situation where a > "checkout-pypi-project" that gained popularity might provide more > incentive for maintainers to provide that metadata and keep it up to > date. As a fallback for projects without that metadata, searching > popular hosting sites like GitHub, BitBucket, GitLab and even > SourceForge, would provide some initial links to investigate. > > Cheers, > Nick. > From tymoteusz.jankowski at gmail.com Sun Mar 13 05:52:46 2016 From: tymoteusz.jankowski at gmail.com (Tymoteusz Jankowski) Date: Sun, 13 Mar 2016 09:52:46 +0000 Subject: [Distutils] [Python-ideas] Version control system link in package meta data In-Reply-To: References: <56E3F313.7060908@sdamon.com> Message-ID: Thanks for information. I think that's enough to me. I'm gonna check all URLs in package data and if anyone match to VCS i clone it. If not fallback to regular package installation. On Sat, Mar 12, 2016 at 3:46 PM Nick Coghlan wrote: > On 12 March 2016 at 20:44, Alexander Walters > wrote: > > I agree, it would be nice if everyone used git (or any of a small set of > > VCS), and all the packages on pypi listed their repositories in the > > metadata. If that were the case, this tool might already exist. In the > > current state of things, though, i don't think it makes much sense to > > produce a general purpose tool for this. > > We don't place any particular requirements on the development > practices of projects publishing their releases through PyPI, so > there's no requirement for a public VCS URL to even exist for a > project, let alone for it to be mentioned in the project metadata. > > That said, since project URLs do make it possible for projects to > share that metadata if they want to, this is a situation where a > "checkout-pypi-project" that gained popularity might provide more > incentive for maintainers to provide that metadata and keep it up to > date. As a fallback for projects without that metadata, searching > popular hosting sites like GitHub, BitBucket, GitLab and even > SourceForge, would provide some initial links to investigate. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 14 04:18:41 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 14 Mar 2016 18:18:41 +1000 Subject: [Distutils] Build systems & working directories Message-ID: I'm currently trying out flit for a new project, and find it nicely simple to use. However, I've also found a current constraint in terms of the source tree layout: if you use a repo structure with a top level "src" directory, then the flit.ini file needs to be in that directory. This means that in PEP 516 terms, the build command will need to be "flit -f src/flit.init" rather than just a plain "flit". I think the PEP could stand to be a bit more explicit that when the specified build_command is invoked, the current working directory will be the root of the source tree. (The PEP sort of implies this already, but I didn't see it explicitly stated anywhere) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Tue Mar 15 12:41:18 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 15 Mar 2016 09:41:18 -0700 Subject: [Distutils] What's up with PyPi maintenance? Message-ID: Probably not the right list but: There are a number of folks having issue siwth new PyPi pacakges not being found by search: https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-search clearly issues have been posted to the project, but has been going on a for a while, with seemingly no action, or response from PyPi developers/maintainers. a) Is there somewhere else we can ping the right fols? b) Is there a serious lack of folks available to address such issues? if (b), should we make a concerted effort to recruit new folks to assist PyPi maintenance / development??? Thanks, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertc at robertcollins.net Tue Mar 15 14:34:48 2016 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 16 Mar 2016 07:34:48 +1300 Subject: [Distutils] abstract build system approaches redux In-Reply-To: References: Message-ID: Just to set expectations: this whole process seems stalled to me; I'm going to context switch and focus on things that can move forward. Someone please ping me when its relevant to put effort in again :). -Rob On 4 March 2016 at 03:11, Nick Coghlan wrote: > On 3 March 2016 at 23:44, Donald Stufft wrote: >> * We make it easier to use a non Python based build system. This is also a nice >> thing, however I don't think it should be a major decider in what API we >> provide. Any reasonable build system is going to have to be available via >> ``pip install`` in some fashion, so even if you write your build system in >> Go or Rust, you're going to have to create a Python package for it anyways, >> and if you're doing that, adding a tiny shim is pretty trivial, something >> like: >> >> import os.path >> import subprocess >> >> BIN = os.path.join(os.path.dirname(__file__), "mytool") >> >> def some_api_function(*args, **kwargs): >> flags = convert_args_kwargs_to_flags(*args, **kwargs) >> subprocess.run(BIN, *flags, check=True) >> >> I don't believe it to be a substantial burden to need to write a tiny wrapper >> if you're going to do something which I believe is going to be very unlikely. > > Ah, you're right, I hadn't accounted for the fact the same shim that > makes a non-Python tool installable as a build dependency could also > handle the adaptation from a Python API to a CLI or FFI based > approach, so putting the standardised interface in Python doesn't > raise any insurmountable barriers to cross-language interoperability - > they just move the additional complexity to the less common case. > > Given that, I'm going to go back to reserving judgement on *all* of > the points Robert mentioned, at least until you've had a chance to > finish writing up your own proposal - the determining factor I thought > I had found turned out not to be so determining after all :) > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -- Robert Collins Distinguished Technologist HP Converged Cloud From brett at python.org Wed Mar 16 12:57:58 2016 From: brett at python.org (Brett Cannon) Date: Wed, 16 Mar 2016 16:57:58 +0000 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: References: Message-ID: On Tue, 15 Mar 2016 at 09:42 Chris Barker wrote: > Probably not the right list but: > > There are a number of folks having issue siwth new PyPi pacakges not being > found by search: > > > https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-search > > clearly issues have been posted to the project, but has been going on a > for a while, with seemingly no action, or response from PyPi > developers/maintainers. > Donald can correct me where I'm wrong, but from my understanding ... > > a) Is there somewhere else we can ping the right fols? > This is as good as any. > > b) Is there a serious lack of folks available to address such issues? > Yes. > > if (b), should we make a concerted effort to recruit new folks to assist > PyPi maintenance / development??? > There are been a couple of calls for help with Warehouse which will replace PyPI as soon as it's finished: https://github.com/pypa/warehouse (the current PyPI code base is really antiquated and from my understanding a real burden to work with at this point). -Brett > > Thanks, > > -CHB > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Mar 16 13:23:53 2016 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 16 Mar 2016 12:23:53 -0500 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: References: Message-ID: from the source, IIUC, warehouse wraps celery for Tasks: https://github.com/pypa/warehouse/blob/master/warehouse/celery.py route to view function mappings are in https://github.com/pypa/warehouse/blob/master/warehouse/routes.py packaging models are in https://github.com/pypa/warehouse/blob/master/warehouse/packaging/models.py warehouse elasticsearch config is in https://github.com/pypa/warehouse/blob/master/warehouse/search.py package search functionality (cp from sqlalchemy to elasticsearch) is in https://github.com/pypa/warehouse/blob/master/warehouse/packaging/search.py https://github.com/pypa/warehouse/blob/master/tests/unit/packaging/test_search.py You can/could re-index the whole collection for search w/ the CLI: https://github.com/pypa/warehouse/blob/master/warehouse/cli/search/reindex.py Fastly downstream HTTP cache invalidation (an async Celery task) is in: https://github.com/pypa/warehouse/blob/master/warehouse/cache/origin/fastly.py https://github.com/pypa/warehouse/blob/master/tests/unit/cache/origin/test_fastly.py IIUC, @Donald @dstufft, there should be a call to ProjectDocType.from_db (either as a celery task or a sqla hook) somewhere that I'm unable to find with GitHub search for 'from_db' On Mar 16, 2016 11:58 AM, "Brett Cannon" wrote: > > > On Tue, 15 Mar 2016 at 09:42 Chris Barker wrote: > >> Probably not the right list but: >> >> There are a number of folks having issue siwth new PyPi pacakges not >> being found by search: >> >> >> https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-search >> >> clearly issues have been posted to the project, but has been going on a >> for a while, with seemingly no action, or response from PyPi >> developers/maintainers. >> > > Donald can correct me where I'm wrong, but from my understanding ... > > >> >> a) Is there somewhere else we can ping the right fols? >> > > This is as good as any. > > >> >> b) Is there a serious lack of folks available to address such issues? >> > > Yes. > > >> >> if (b), should we make a concerted effort to recruit new folks to assist >> PyPi maintenance / development??? >> > > There are been a couple of calls for help with Warehouse which will > replace PyPI as soon as it's finished: https://github.com/pypa/warehouse (the > current PyPI code base is really antiquated and from my understanding a > real burden to work with at this point). > > -Brett > > >> >> Thanks, >> >> -CHB >> >> >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker at noaa.gov >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Mar 16 18:15:58 2016 From: richard at python.org (Richard Jones) Date: Thu, 17 Mar 2016 09:15:58 +1100 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: References: Message-ID: On 16 March 2016 at 03:41, Chris Barker wrote: > Probably not the right list but: > > There are a number of folks having issue siwth new PyPi pacakges not being > found by search: > > > https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-search > > clearly issues have been posted to the project, but has been going on a > for a while, with seemingly no action, or response from PyPi > developers/maintainers. > > a) Is there somewhere else we can ping the right fols? > > b) Is there a serious lack of folks available to address such issues? > This is the correct answer. I can't even keep on top of the request from people wanting to take over packages. if (b), should we make a concerted effort to recruit new folks to assist > PyPi maintenance / development??? > That would be nice, but would be pointless until Warehouse goes live. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 17 20:51:08 2016 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 17 Mar 2016 17:51:08 -0700 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: References: Message-ID: <-6652552687042308282@unknownmsgid> b) Is there a serious lack of folks available to address such issues? > This is the correct answer. I can't even keep on top of the request from people wanting to take over packages. if (b), should we make a concerted effort to recruit new folks to assist > PyPi maintenance / development??? > That would be nice, but would be pointless until Warehouse goes live. When do we expect that? A lot of people rely on the current system, we really need to find a way to maintain it 'till it's replaced. CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Mar 18 12:16:23 2016 From: brett at python.org (Brett Cannon) Date: Fri, 18 Mar 2016 16:16:23 +0000 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: <-6652552687042308282@unknownmsgid> References: <-6652552687042308282@unknownmsgid> Message-ID: On Thu, 17 Mar 2016 at 17:51 Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > > b) Is there a serious lack of folks available to address such issues? >> > > This is the correct answer. I can't even keep on top of the request from > people wanting to take over packages. > > > if (b), should we make a concerted effort to recruit new folks to assist >> PyPi maintenance / development??? >> > > That would be nice, but would be pointless until Warehouse goes live. > > > When do we expect that? A lot of people rely on the current system, we > really need to find a way to maintain it 'till it's replaced. > Do realize that this is in regards to the search box, not the ability for PyPI to keep standing to serve files. Obviously if something happened to file serving or some critical aspect of PyPI then that would get addressed ASAP. But fixing a search box is not critical infrastructure so I don't think it warrants a "all hands on deck" call for help over trying to get the replacement code done instead (I get people use the search box, but there are well-known alternatives for searching PyPI). In all honesty, if people want a functioning search box on PyPI I would say they should submit a patch to use https://cse.google.com/cse/ . -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Mar 18 16:11:29 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 18 Mar 2016 13:11:29 -0700 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: References: <-6652552687042308282@unknownmsgid> Message-ID: On Fri, Mar 18, 2016 at 9:16 AM, Brett Cannon wrote: > When do we expect that? A lot of people rely on the current system, we >> really need to find a way to maintain it 'till it's replaced. >> > > Do realize that this is in regards to the search box, not the ability for > PyPI to keep standing to serve files. Obviously if something happened to > file serving or some critical aspect of PyPI then that would get addressed > ASAP. But fixing a search box is not critical infrastructure so I don't > think it warrants a "all hands on deck" call for help > well, it's not just the search box, it's also "pip search" from the command line. And sure -- still not an all hands on deck, but maybe a note on the issue that the team is aware of the issue, but too understaffed and focused on the new and better to go in and fix it. just today, another note on the issue: " Is the issue even acknowledged?" https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-search This really gives the impression of a non-maintained system. I'm not trying to be critical of a volunteer effort here -- but maybe we really do need a call for help out there. Honestly, I, for one, am poorly qualified, but it's likely there is someone out there with the requisite skills tha might want to help. -CHB > over trying to get the replacement code done instead (I get people use the > search box, but there are well-known alternatives for searching PyPI). > like pip search? also apparently broken? > In all honesty, if people want a functioning search box on PyPI I would > say they should submit a patch to use https://cse.google.com/cse/ . > Maybe -- actually, it turns out a plain old google search for "PyPi package name" seems to work, once you've given Google a chance to index it. So in the meantime maybe the search box should be removed in favor of a note suggesting a google search -- or a link that does a google search for you. A broken search is a really bad thing to have on a web site. I'm sure I'm not the only one that thought that the fact that search didn't find it meant that that it hadn't been uploaded properly. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Mar 18 16:29:36 2016 From: donald at stufft.io (Donald Stufft) Date: Fri, 18 Mar 2016 16:29:36 -0400 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: <-6652552687042308282@unknownmsgid> References: <-6652552687042308282@unknownmsgid> Message-ID: <67D5E3F3-F6A7-4D33-B2EC-FCCD71D32469@stufft.io> > On Mar 17, 2016, at 8:51 PM, Chris Barker - NOAA Federal wrote: > > When do we expect that? A lot of people rely on the current system, we really need to find a way to maintain it 'till it's replaced. Practically speaking I?m the only person maintaining the actual code behind legacy PyPI right now. Due to some personal stuff I haven?t had the mental bandwidth to dig through debugging the ball of twine that is legacy PyPI. It?s been sort of falling apart a bit and fixing some of these issues require a substantial amount of effort. I don?t know what?s wrong with search, it?s probably not one of those kinds of issues, but the hope is that we can just squash a bunch of these at once with the new code base. As far as ?When do we expect that?, the answer is basically ?when it?s ready?. You can see what?s done so far at warehouse.python.org (backed by the same live database) and the current work is being done at github.com/pypa/warehouse . ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From alex.gronholm at nextday.fi Fri Mar 18 17:10:40 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Fri, 18 Mar 2016 23:10:40 +0200 Subject: [Distutils] Wheels and Python implementation markers Message-ID: <56EC6ED0.1060904@nextday.fi> What is the "correct" way to specify (for building wheels) in setup.py that a dependency should only be installed on CPython? If so, how should I go about it? Nothing I've tried seems to work right. The python_implementation marker worked up until packaging 16.5 and then platform_python_implementation does not work with older versions of pip. From noah at coderanger.net Fri Mar 18 19:16:16 2016 From: noah at coderanger.net (Noah Kantrowitz) Date: Fri, 18 Mar 2016 16:16:16 -0700 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: <67D5E3F3-F6A7-4D33-B2EC-FCCD71D32469@stufft.io> References: <-6652552687042308282@unknownmsgid> <67D5E3F3-F6A7-4D33-B2EC-FCCD71D32469@stufft.io> Message-ID: <9872D5EC-2FC4-494A-9159-1A9F53110063@coderanger.net> > On Mar 18, 2016, at 1:29 PM, Donald Stufft wrote: > > >> On Mar 17, 2016, at 8:51 PM, Chris Barker - NOAA Federal wrote: >> >> When do we expect that? A lot of people rely on the current system, we really need to find a way to maintain it 'till it's replaced. > > Practically speaking I?m the only person maintaining the actual code behind legacy PyPI right now. Due to some personal stuff I haven?t had the mental bandwidth to dig through debugging the ball of twine that is legacy PyPI. It?s been sort of falling apart a bit and fixing some of these issues require a substantial amount of effort. I don?t know what?s wrong with search, it?s probably not one of those kinds of issues, but the hope is that we can just squash a bunch of these at once with the new code base. > > As far as ?When do we expect that?, the answer is basically ?when it?s ready?. You can see what?s done so far at warehouse.python.org (backed by the same live database) and the current work is being done at github.com/pypa/warehouse. Just to make it explicit, if anyone wants to submit a patch with a cogent explanation of what was wrong I'm sure it would get fixed ASAP. Such is the eternal rallying cry of FOSS :-) --Noah -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.barker at noaa.gov Fri Mar 18 19:35:10 2016 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 18 Mar 2016 16:35:10 -0700 Subject: [Distutils] What's up with PyPi maintenance? In-Reply-To: <9872D5EC-2FC4-494A-9159-1A9F53110063@coderanger.net> References: <-6652552687042308282@unknownmsgid> <67D5E3F3-F6A7-4D33-B2EC-FCCD71D32469@stufft.io> <9872D5EC-2FC4-494A-9159-1A9F53110063@coderanger.net> Message-ID: On Fri, Mar 18, 2016 at 4:16 PM, Noah Kantrowitz wrote: > Just to make it explicit, if anyone wants to submit a patch with a cogent > explanation of what was wrong I'm sure it would get fixed ASAP. Good to know -- if issues aren't commented on, it's hard to know if a PR will get reviewed, either. -CHB > Such is the eternal rallying cry of FOSS :-) > > --Noah > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From crusoe at ucdavis.edu Mon Mar 21 07:40:51 2016 From: crusoe at ucdavis.edu (Michael R. Crusoe) Date: Mon, 21 Mar 2016 11:40:51 +0000 Subject: [Distutils] python_version>="2.7" environment marker difficulties Message-ID: Hello, To support a dependency[0] which has a different name for its Py2 & Py3 versions, our program schema_salad uses the "python_version" environment marker. This seems to not be supported when users have older setuptools installed[1] With setuptools before version 17.1 one gets error in schema-salad setup command: Invalid environment marker: python_version>="2.7" I tried to bundle a newer version with ez_setup.py but that isn't working either: https://github.com/common-workflow-language/schema_salad/commit/04a956dbef40b32ef9c486ddab4dcff544dc2c65#commitcomment-16745845 "ez_setup() bootstraps transparently if you're using python setup.py install but not for pip install of the same package (where it complains that you need to upgrade setuptools first)" Code is at https://github.com/common-workflow-language/schema_salad/blob/envmarker-test/setup.py#L45 Is there a better way to deal with the divergent names for our dependency? Thanks in advance, [0] https://pypi.python.org/pypi/avro vs https://pypi.python.org/pypi/avro-python3 [1] https://github.com/common-workflow-language/schema_salad/issues/17#issue-141147041 -- Michael R. Crusoe CWL Community Engineer crusoe at ucdavis.edu Common Workflow Language project University of California, Davis https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe -------------- next part -------------- An HTML attachment was scrubbed... URL: From ginofontanilla at gmail.com Sun Mar 20 20:35:20 2016 From: ginofontanilla at gmail.com (Gene Fontanilla) Date: Mon, 21 Mar 2016 08:35:20 +0800 Subject: [Distutils] Python on AIX 7.1 Message-ID: Can anyone guide me in installing Python AIX? i have no experience in AIX. Thanks and Regards, -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Mon Mar 21 09:16:01 2016 From: cournape at gmail.com (David Cournapeau) Date: Mon, 21 Mar 2016 13:16:01 +0000 Subject: [Distutils] PEP 425 vs pip In-Reply-To: References: Message-ID: On Fri, Mar 11, 2016 at 3:26 PM, Nick Coghlan wrote: > On 11 March 2016 at 19:41, David Cournapeau wrote: > > Hi, > > > > At Enthought we have been trying to use PEP 425 tags to include its > metadata > > in our packages. It has been harder than expected because of missing > > information in PEP 425, and maybe inconsistencies with what pip actually > > does. > > > > I think PEP 425 should be updated to take into account pip actual > > implementation: > > > > 1. python tag and pypy: the PEP says "the version is py_version_nodot . > > CPython gets away with no dot, but if one is needed the underscore _ is > used > > instead. PyPy should probably use its own versions here pp18 , pp19 ". > This > > is not what pip does as of version 8.1.0 for pypy. Instead, the version > is > > python major + sys.pypy_version_info.major + sys.pypy_version_info.minor. > > > > 2. It would be nice to know how to deal with other alternative > > implementations as well. Would the scheme in 1. work for jython, etc... ? > > > > 3. The abi tag should at least be specified for cpython IMO. E.g. the > > meaning of `m`, `u` and `d` should be clearly specified. > > > > While the PEP mentions each implementation community is welcome to use > the > > abi tag as they wish, given that in practice it is defined in pip, I > think > > that should be specified as well. > > > > Do the above make sense ? If so, I could spend some time to update it (or > > create a new amending PEP ?) > > Proposed errrata & minor admentments for PEP 425 can be submitted as a > PyPUG issue with a PR against > > https://packaging.python.org/en/latest/specifications/#platform-compatibility-tags > > (the actual technical details of the proposed changes should still be > discussed here) > Up: would it help doing the PR first as a basic for discussion ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at ionelmc.ro Mon Mar 21 09:26:21 2016 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Mon, 21 Mar 2016 15:26:21 +0200 Subject: [Distutils] python_version>="2.7" environment marker difficulties In-Reply-To: References: Message-ID: I think most users will understand that they need to upgrade their pip/setuptools. If they complain about Ubuntu not shipping recent tooling or whatever then just send them to the Ubuntu bug tracker :-) Regarding your issue, since you don't depend on minor python version you could easily publish one wheel for py2 and one for py3. This won't need env markers and will avoid wheel poisoning issues, in most cases at least. But this complicates the release process - is it worth it? Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro On Mon, Mar 21, 2016 at 1:40 PM, Michael R. Crusoe wrote: > Hello, > > To support a dependency[0] which has a different name for its Py2 & Py3 > versions, our program schema_salad uses the "python_version" environment > marker. > > This seems to not be supported when users have older setuptools > installed[1] > > With setuptools before version 17.1 one gets > > error in schema-salad setup command: Invalid environment marker: > python_version>="2.7" > > I tried to bundle a newer version with ez_setup.py but that isn't working > either: > https://github.com/common-workflow-language/schema_salad/commit/04a956dbef40b32ef9c486ddab4dcff544dc2c65#commitcomment-16745845 > > "ez_setup() bootstraps transparently if you're using python setup.py > install but not for pip install of the same package (where it complains > that you need to upgrade setuptools first)" > > Code is at > https://github.com/common-workflow-language/schema_salad/blob/envmarker-test/setup.py#L45 > > Is there a better way to deal with the divergent names for our dependency? > > Thanks in advance, > > [0] https://pypi.python.org/pypi/avro vs > https://pypi.python.org/pypi/avro-python3 > [1] > https://github.com/common-workflow-language/schema_salad/issues/17#issue-141147041 > > -- > Michael R. Crusoe CWL Community Engineer crusoe at ucdavis.edu > Common Workflow Language project University of California, Davis > https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Mon Mar 21 12:04:54 2016 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Mon, 21 Mar 2016 11:04:54 -0500 Subject: [Distutils] Python on AIX 7.1 In-Reply-To: References: Message-ID: On Sun, Mar 20, 2016 at 7:35 PM, Gene Fontanilla wrote: > Can anyone guide me in installing Python AIX? i have no experience in AIX. This mailing list is about distributing python packages, not python itself. You might have better luck asking on the general python-list. From ncoghlan at gmail.com Wed Mar 23 06:44:00 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Mar 2016 20:44:00 +1000 Subject: [Distutils] PEP 425 vs pip In-Reply-To: References: Message-ID: On 21 March 2016 at 23:16, David Cournapeau wrote: > On Fri, Mar 11, 2016 at 3:26 PM, Nick Coghlan wrote: >> Proposed errrata & minor admentments for PEP 425 can be submitted as a >> PyPUG issue with a PR against >> >> https://packaging.python.org/en/latest/specifications/#platform-compatibility-tags >> >> (the actual technical details of the proposed changes should still be >> discussed here) > > Up: would it help doing the PR first as a basic for discussion ? Yep, having something lighter than a PEP to track the specifics of proposed changes was one of the key intended benefits of changing the process. Cheers, Nick. P.S. Note I'll be on vacation and deliberately laptop free for a week starting Friday though. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From demianbrecht at gmail.com Fri Mar 25 19:17:16 2016 From: demianbrecht at gmail.com (Demian Brecht) Date: Fri, 25 Mar 2016 16:17:16 -0700 Subject: [Distutils] Processing dependencies on Gihtub Message-ID: <8C2E7F71-501B-4BC1-AB7C-C0588AC2995A@gmail.com> Hi all, I'm investigating the ability to replace PyPI with Github-based mirrors. The use case is that there is no outgoing internet access from a set of hosts responsible for building Python packages. However, there /is/ access to a Github enterprise instance. Installing top level packages through git+https works great, but (unless I'm wrong) falls down when handling dependencies. I've looked at --find-links, which doesn't seem to do the trick (I'm assuming it's expecting a PyPI-compatible index). I've also read the --process-dependency-links is deprecated and will eventually be removed. Is that correct? If that's all accurate, are there any suggested paths to install top level packages and their respective dependencies solely using Github? Are there any API or HTML pages that I'm unaware of that would conform to pip requirements? The only avenue that I can think of is to write a small proxy service that simply presents a format understood by pip. One of the things that I'm trying to avoid is touching mirrored projects. Not all projects use requirements files and obviously none of them will be using the internal GHE instance that I'm looking to use as a home for mirrored projects. Thanks, ------- GPG Fingerprint: 9530 B4AF 551B F3CD A45C 476C D4E5 662D DB97 69E3 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 842 bytes Desc: Message signed with OpenPGP using GPGMail URL: From p.f.moore at gmail.com Sat Mar 26 12:03:28 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 26 Mar 2016 16:03:28 +0000 Subject: [Distutils] Processing dependencies on Gihtub In-Reply-To: <8C2E7F71-501B-4BC1-AB7C-C0588AC2995A@gmail.com> References: <8C2E7F71-501B-4BC1-AB7C-C0588AC2995A@gmail.com> Message-ID: On 25 March 2016 at 23:17, Demian Brecht wrote: > If that's all accurate, are there any suggested paths to install top level packages and their respective dependencies solely using Github? Are there any API or HTML pages that I'm unaware of that would conform to pip requirements? The only avenue that I can think of is to write a small proxy service that simply presents a format understood by pip If you have http access to github project pages, you could host your own PyPI mirror as a set of github pages - see https://packaging.python.org/en/latest/self_hosted_repository/ Paul From nicholas.chammas at gmail.com Sun Mar 27 21:11:11 2016 From: nicholas.chammas at gmail.com (Nicholas Chammas) Date: Mon, 28 Mar 2016 01:11:11 +0000 Subject: [Distutils] Thank you for the ability to do `pip install git+https://...` Message-ID: Dunno how old/new this feature is, or what people did before it existed, but I just wanted to thank the people who thought of and built the ability to do installs from git+https. It lets me offer the following to my users when they want the ?bleeding edge? version of my project: pip install git+https://github.com/nchammas/flintrock I also use this capability to install and test contributors? branches when they open PRs against my project. For example: pip install git+https://github.com/contributor/flintrock at branch It?s a great feature and makes my work a bit easier. Thank you for building it. I?m still waiting for when I can give the PyPA some money for all the good and sorely needed work that y?all do? Anyway, keep it up. Nick ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.gronholm at nextday.fi Mon Mar 28 08:54:20 2016 From: alex.gronholm at nextday.fi (=?UTF-8?Q?Alex_Gr=c3=b6nholm?=) Date: Mon, 28 Mar 2016 15:54:20 +0300 Subject: [Distutils] Thank you for the ability to do `pip install git+https://...` In-Reply-To: References: Message-ID: <56F9297C.9050402@nextday.fi> You could always point pip to the automatically generated zip (https://github.com/nchammas/flintrock/archive/master.zip). That way you won't have to wait for git to pointlessly download your entire version history when you just want to install something. 28.03.2016, 04:11, Nicholas Chammas kirjoitti: > > Dunno how old/new this feature is, or what people did before it > existed, but I just wanted to thank the people who thought of and > built the ability to do installs from |git+https|. > > It lets me offer the following to my users when they want the > ?bleeding edge? > version of > my project: > > |pip install git+https://github.com/nchammas/flintrock | > > I also use this capability to install and test contributors? branches > when they open PRs against my project. For example: > > |pip install git+https://github.com/contributor/flintrock at branch | > > It?s a great feature and makes my work a bit easier. Thank you for > building it. > > I?m still waiting for when I can give the PyPA some money > for all the good and > sorely needed work that y?all do? > > Anyway, keep it up. > > Nick > > ? > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Wed Mar 30 20:03:49 2016 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 30 Mar 2016 17:03:49 -0700 Subject: [Distutils] question re: local development workflow / egg-info / synced directories Message-ID: Hi, I have a local development workflow question I'm looking for feedback or suggestions on. It relates to installing things in editable mode in conjunction with syncing source files to a VM. I'm developing a Python project that is deployed / runs inside a Docker container. I do my local development on a Mac, and I use the following script to "sync" my local source files to the Docker container: https://github.com/brikis98/docker-osx-dev At Docker build time, the project source files are installed in "editable" mode, which creates egg-info directories inside the source directories in the Docker image. When developing locally, however, the sync process mounts the synced directory "over" the directory that was initialized at Docker-build time. Thus the egg-info directories are missing from the synced version of the directory. Does anyone have any suggestions for handling this scenario? The three possibilities I can think of are-- 1. re-install the projects in the Docker container in editable mode after mounting the synced directory (which is a bit hacky because there is no obvious "hook" with the tools I'm using), 2. install the projects in editable mode locally on my Mac so the egg-info directories will also sync over (but I'm not sure if egg-info directories are sufficiently cross-platform), or 3. maybe there is a way to install things in "editable" mode that doesn't require egg-info directories to be written alongside the source files? Does anyone have any suggestions? And does anyone know if (3) is possible? Thanks a lot for any help, --Chris From contact at ionelmc.ro Thu Mar 31 01:56:19 2016 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 31 Mar 2016 08:56:19 +0300 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On Thu, Mar 31, 2016 at 3:03 AM, Chris Jerdonek wrote: > When developing locally, however, the sync process mounts the synced > directory "over" the directory that was initialized at Docker-build > time. Thus the egg-info directories are missing from the synced > version of the directory. > ?Are you mounting the site-packages or something else?? > 2. install the projects in editable mode locally on my Mac so the > egg-info directories will also sync over (but I'm not sure if egg-info > directories are sufficiently cross-platform), or > ?Assuming you mount the site-packages this should be fine as long as the paths are the same, this is because easy-install.pth will contain absolute paths?. When you do `setup.py develop` an entry is added in easy-install.pth and that egg-info dir is created. Note that the egg-info metadata is mostly used for entrypoint discovery (eg: you use console_scripts or plugin system based on entrypoints). Also, develop installs allow overriding the develop command, so they might do something extra (like pytest-cov "installs" a pth file for the subprocess support). That means you can just symlink or use a pth file if you don't care about any of those features. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Thu Mar 31 02:22:52 2016 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 30 Mar 2016 23:22:52 -0700 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On Wed, Mar 30, 2016 at 10:56 PM, Ionel Cristian M?rie? wrote: > > On Thu, Mar 31, 2016 at 3:03 AM, Chris Jerdonek > wrote: >> >> When developing locally, however, the sync process mounts the synced >> directory "over" the directory that was initialized at Docker-build >> time. Thus the egg-info directories are missing from the synced >> version of the directory. > > Are you mounting the site-packages or something else? No, I'm only mounting the project / source files. The Docker build step creates a virtualenv for use inside the image, which is in a location different from the synced directory. I'm not syncing site-packages as I'm not installing the requirements on my Mac (and in particular the "downloadable" requirements). I'm only installing them inside the Docker image at Docker build time (and it's not clear to me whether packages pip-installed on my Mac would even be compatible with a different OS on the Docker image). It would be nice if pip-installing a project directory in editable mode didn't require writing to the project directory. Is there a fundamental reason it needs to? Thanks, --Chris >> 2. install the projects in editable mode locally on my Mac so the >> egg-info directories will also sync over (but I'm not sure if egg-info >> directories are sufficiently cross-platform), or > > > Assuming you mount the site-packages this should be fine as long as the > paths are the same, this is because easy-install.pth will contain absolute > paths. When you do `setup.py develop` an entry is added in easy-install.pth > and that egg-info dir is created. > > Note that the egg-info metadata is mostly used for entrypoint discovery (eg: > you use console_scripts or plugin system based on entrypoints). > > Also, develop installs allow overriding the develop command, so they might > do something extra (like pytest-cov "installs" a pth file for the subprocess > support). > > That means you can just symlink or use a pth file if you don't care about > any of those features. > > > > > Thanks, > -- Ionel Cristian M?rie?, http://blog.ionelmc.ro From contact at ionelmc.ro Thu Mar 31 05:00:05 2016 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 31 Mar 2016 12:00:05 +0300 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek wrote: > It would be nice if pip-installing a project directory in editable > mode didn't require writing to the project directory. Is there a > fundamental reason it needs to? > ?I'd speculate that no one was annoyed enough to fix it.? You could manually create the egg-info (setup.py egg_info --egg-base=somewhere) and do the linking manually. I'm preeeety sure my suggestion will raise some eyebrows here :) Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Mar 31 05:12:56 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 10:12:56 +0100 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On 31 March 2016 at 10:00, Ionel Cristian M?rie? wrote: > On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek > wrote: >> >> It would be nice if pip-installing a project directory in editable >> mode didn't require writing to the project directory. Is there a >> fundamental reason it needs to? > > > I'd speculate that no one was annoyed enough to fix it. Editable mode is basically managed by setuptools. So any fix would need to be in setuptools, not pip. It's quite possible that the reason editable mode doesn't get much love is that from a user's perspective it seems like a pip issue, but the pip devs see it as a setuptools problem they can't do much about. Not many users have the time or inclination to follow up on a bug report with the downstream project, so things tend to stay as they are. Paul From chris.jerdonek at gmail.com Thu Mar 31 06:20:45 2016 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 31 Mar 2016 03:20:45 -0700 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On Thu, Mar 31, 2016 at 2:12 AM, Paul Moore wrote: > On 31 March 2016 at 10:00, Ionel Cristian M?rie? wrote: >> On Thu, Mar 31, 2016 at 9:22 AM, Chris Jerdonek >> wrote: >>> >>> It would be nice if pip-installing a project directory in editable >>> mode didn't require writing to the project directory. Is there a >>> fundamental reason it needs to? >> >> I'd speculate that no one was annoyed enough to fix it. > > Editable mode is basically managed by setuptools. So any fix would > need to be in setuptools, not pip. After your two comments, I searched setuptools's tracker and found the following issue ("develop does not support placing the egg outside of the source folder"): https://github.com/pypa/setuptools/issues/361 So it looks like the feature may already be present in setuptools, but just not working correctly (i.e. the feature has a bug). If this feature were working in setuptools, would pip need any changes to "install -e" to allow passing along this extra info to setuptools, or is there already a mechanism to allow pass-through of extra setuptools options? Thanks, --Chris > It's quite possible that the reason > editable mode doesn't get much love is that from a user's perspective > it seems like a pip issue, but the pip devs see it as a setuptools > problem they can't do much about. Not many users have the time or > inclination to follow up on a bug report with the downstream project, > so things tend to stay as they are. > > Paul From contact at ionelmc.ro Thu Mar 31 06:46:16 2016 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 31 Mar 2016 13:46:16 +0300 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On Thu, Mar 31, 2016 at 1:20 PM, Chris Jerdonek wrote: > If this feature were working in setuptools, would pip need any changes > to "install -e" to allow passing along this extra info to setuptools, > or is there already a mechanism to allow pass-through of extra > setuptools options? > Pip install has the --install-option and --global-option? arguments, try your luck with them - they are for fairly advanced usecases. Pip already passes other options so there's risk of breakages or weird issues if you pass conflicting options. Thanks, -- Ionel Cristian M?rie?, http://blog.ionelmc.ro -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Mar 31 07:00:47 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 31 Mar 2016 12:00:47 +0100 Subject: [Distutils] question re: local development workflow / egg-info / synced directories In-Reply-To: References: Message-ID: On 31 March 2016 at 11:46, Ionel Cristian M?rie? wrote: > On Thu, Mar 31, 2016 at 1:20 PM, Chris Jerdonek > wrote: >> >> If this feature were working in setuptools, would pip need any changes >> to "install -e" to allow passing along this extra info to setuptools, >> or is there already a mechanism to allow pass-through of extra >> setuptools options? > > > Pip install has the --install-option and --global-option arguments, try your > luck with them - they are for fairly advanced usecases. Pip already passes > other options so there's risk of breakages or weird issues if you pass > conflicting options. Setuptools also supports the standard setup.cfg mechanism. That may be an easier alternative. Paul