From ncoghlan at gmail.com Sun Dec 1 05:15:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Dec 2013 14:15:49 +1000 Subject: [Distutils] Handling the binary dependency management problem Message-ID: I've had a couple of conversations recently that suggest it's high time I posted this particular idea publicly, so here's my suggestion for dealing with (or, more accurately, avoiding) the binary dependency management problem for upstream distribution in the near term. = Defining the problem = The core packaging standards and tools need to deal with a couple of different use cases. We want them to be usable for beginners and folks that aren't professional developers running Python locally, but we also want them to be suitable for professionally administered server environments, and to integrate reasonably well with downstream redistributors (most notably Linux distributions, but also other redistributors like ActiveState, Enthought and Continuum Analytics). For pure Python modules, these two objectives are easy to reconcile. For simple, self-contained, C extension modules, it gets harder, which is why PyPI only allows wheel files for Mac OS X and Windows, and expects them to be compatible with the binary installers provided on python.org. Resolving this to allow platform specific wheel files on PyPI for Linux distributions, Cygwin, MacPorts, homebrew, etc is going to require some work on better defining the "platform" tag in PEP 425. For arbitrary binary dependencies, however, I contend that reconciling the two different use cases is simply infeasible, as pip and venv have to abide by the following two restrictions: 1. Allow low impact security updates to components low in the stack (e.g. deploying a CPython security update shouldn't require rebuilding any other components) 2. Allow interoperability with a variety of packaging systems, including supporting access to Python modules provided solely through the system package manager (e.g. yum/rpm related modules on Fedora and derived distros) At this level, it's often impractical to offer prebuilt binary components with external dependencies, as you *also* need a mechanism to deliver the external binary dependencies themselves, that platform integrators can override if they want. = Proposed approach = Since supporting both cross-platform external binaries *and* allowing the substition of platform specific binaries in the same tool is difficult, I propose that we don't even try (at least not until we're much further down the packaging improvement road, and have solved all the *other* issues facing the packaging infrastructure) Instead, I suggest that the Python Packaging User Guide *explicitly* recommend a two level solution: 1. For simple cases without complex binary dependencies, and for cases where platform integration and/or low impact security updates are needed, use the core pip/virtualenv toolchain. This may sometimes mean needing to build components with external dependencies from source, and that's OK (it's the price we pay at this level for supporting platform integration). 2. For cross-platform handling of external binary dependencies, we recommend boostrapping the open source conda toolchain, and using that to install pre-built binaries (currently administered by the Continuum Analytics folks). Specifically, commands like the following should work on POSIX systems without needing any local build machinery, and without needing all the projects in the chain to publish wheels: "pip install conda && conda init && conda install ipython" For many end users just running things locally (especially beginners and non-developers), using conda will be the quickest and easiest way to get up and running. For professional developers and administrators, option 1 will provide the finer control and platform interoperability that they often need. conda already has good coverage of the scientific stack, but may need additional contributions to cover other notoriously hard to build components (such as crypto libraries and game development libraries). = What would this mean for the wheel format? = conda has its own binary distribution format, using hash based dependencies. It's this mechanism which allows it to provide reliable cross platform binary dependency management, but it's also the same mechanism that prevents low impact security updates and interoperability with platform provided packages. So wheels would remain the core binary format, and we'd continue to recommend publishing them whenever feasible. However, we'd postpone indefinitely the question of attempting to deal with arbitrary external dependency support. = Blocking issues = Even if we agree this a good way forward in the near term, there are a couple of technical issues that would need to be resolved before making it official in the user guide. There is currently at least one key blocking issue on the conda side, where it breaks if run inside a symlinked virtual environment (which both virtualenv and pyvenv create by default in POSIX environments): https://github.com/ContinuumIO/conda/issues/360 I believe conda also currently conflicts with virtualenvwrapper regarding the meaning of activate/deactivate at the command line. There aren't any blockers I am aware on the pip/virtualenv side, but the fact --always-copy is currently broken in virtualenv prevents a possible workaround for the conda issue above: https://github.com/pypa/virtualenv/issues/495 (pyvenv doesn't offer an --always-copy option, just the option to use symlinks on Windows where they're not used by default due to the associated permissions restrictions, so the primary resolution still needs to be for conda to correctly handle that situation and convert the venv Python to a copy rather than failing) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Dec 1 14:10:42 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 1 Dec 2013 13:10:42 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 1 December 2013 04:15, Nick Coghlan wrote: > 2. For cross-platform handling of external binary dependencies, we > recommend boostrapping the open source conda toolchain, and using that > to install pre-built binaries (currently administered by the Continuum > Analytics folks). Specifically, commands like the following should > work on POSIX systems without needing any local build machinery, and > without needing all the projects in the chain to publish wheels: "pip > install conda && conda init && conda install ipython" Hmm, this is a somewhat surprising change of direction. You mention POSIX here - but do you intend this to be the standard approach on Windows too? Just as a test, I tried the above, on Python 3.3 on Windows 64-bit. This is python.org python, installed in a virtualenv. I'm just going off what you said above - if there are more explicit docs, I can try using them (but I *don't* want to follow the "official" Anaconda docs, as they talk about using Anaconda python, and about using conda to manage environments, rather than virtualenv). pip install conda worked OK, but it installed a pure-Python version of PyYAML (presumably because the C accelerator needs libyaml, so can't be built without a bit of extra work - that's a shame but see below). conda init did something, no idea what but it seemed to be fine. conda install ipython then worked, it seems to have installed a binary version of pyyaml. Then, however, yaml install numpy fails: >conda install numpy failed to create process. It looks like the binary yaml module is broken. Doing "import yaml" in a python session gives a runtime error "An application has made an attempt to load the C runtime library incorrectly". I can report this as a bug to conda, I guess (I won't, because I don't know where to report conda bugs, and I don't expect to have time to find out or help diagnose the issues when the developers investigate - it was something I tried purely for curiosity). But I wouldn't be happy to see this as the recommended approach until it's more robust than this. Paul From vinay_sajip at yahoo.co.uk Sun Dec 1 16:04:02 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 1 Dec 2013 15:04:02 +0000 (GMT) Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: Message-ID: <1385910242.34561.YahooMailBasic@web171404.mail.ir2.yahoo.com> On Sun, 1/12/13, Nick Coghlan wrote: > (pyvenv doesn't offer an --always-copy option, just the option to use symlinks on It does - you should be able to run pyvenv with --copies to force copying, even on POSIX. Regards, Vinay Sajip From oscar.j.benjamin at gmail.com Sun Dec 1 16:22:26 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 1 Dec 2013 15:22:26 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Dec 1, 2013 1:10 PM, "Paul Moore" wrote: > > On 1 December 2013 04:15, Nick Coghlan wrote: > > 2. For cross-platform handling of external binary dependencies, we > > recommend boostrapping the open source conda toolchain, and using that > > to install pre-built binaries (currently administered by the Continuum > > Analytics folks). Specifically, commands like the following should > > work on POSIX systems without needing any local build machinery, and > > without needing all the projects in the chain to publish wheels: "pip > > install conda && conda init && conda install ipython" > > Hmm, this is a somewhat surprising change of direction. Indeed it is. Can you clarify a little more how you've come to this conclusion, Nick and perhaps explain what conda is? I looked at conda some time ago and it seemed to be aimed at HPC (high performance computing) clusters which is a niche use case where you have large networks of computation nodes containing identical hardware. (unless I'm conflating it with something else). Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sun Dec 1 20:21:57 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 1 Dec 2013 11:21:57 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > For arbitrary binary dependencies, however, I contend that reconciling > the two different use cases is simply infeasible, as pip and venv have > to abide by the following two restrictions: > To be clear, what's a good example of a common non-science PyPI package that has an "arbitrary binary dependency"? psycopg2? > For many end users just running things locally (especially beginners > and non-developers), using conda will be the quickest and easiest way > to get up and running. Conda/Anaconda is an alien world right now to most non-science people (including me) Working in an alien world, is never the "quickest" or "easiest" way at first, but I'm curious to try. Some PyPA people actually need to try using it for real, and get comfortable with it. >> sometimes mean needing to build components with external dependencies from source you mean build once (or maybe after system updates for wheels with external binary deps), and cache as a local wheel, right? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Dec 1 21:48:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 1 Dec 2013 20:48:24 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 1 December 2013 19:21, Marcus Smith wrote: >>> sometimes mean needing to build components with external dependencies >>> from source > > you mean build once (or maybe after system updates for wheels with external > binary deps), and cache as a local wheel, right? Note that it is possible to convert binary packages from other formats to wheel. We already support egg and bdist_wininst. If conda has built packages that are worth supporting, I doubt that a conda-to-wheel converter would be hard to write. I'd be willing to write one if someone can point me at a spec for the conda package format (I couldn't find one from a brief look through the docs). Conversely, if I have a wininst installer, a wheel or an egg, is there a converter to conda format? I can't see having to use conda for some things and pip for others as being a move towards lessening user confusion. I see conda (and enthought) as more like distributions - they sit in the same space as rpms and debs. I don't see them as alternatives to pip. Am I wrong? After all, both conda and enthought supply Python interpreter builds as well as package repositories. And both feel like unified ecosystems (you manage everything Python-related with conda) rather than tools like pip (that works with existing Python standards and interoperates with easy_install, distutils, etc). If the issue is simply around defining compatibility tags that better describe the various environments around, then let's just get on with that - we're going to have to do it in the end anyway, why temporarily promote an alternative solution just to change our recommendation later? Paul From vinay_sajip at yahoo.co.uk Sun Dec 1 23:07:37 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 1 Dec 2013 22:07:37 +0000 (GMT) Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: Message-ID: <1385935657.29586.YahooMailBasic@web171401.mail.ir2.yahoo.com> On Sun, 1/12/13, Paul Moore wrote: > If the issue is simply around defining compatibility tags that better > describe the various environments around, then let's just get on with > that - we're going to have to do it in the end anyway, why temporarily > promote an alternative solution just to change our recommendation > later? This makes sense to me. We should refine the compatibility tags as much as is required. It would be nice if there was some place (on PyPI, or elsewhere) where users could request binary distributions for specific packages for particular environments, and then some kind people with those environments might be able to build those wheels and upload them ... a bit like Christoph Gohlke does for Windows. Regards, Vinay Sajip From ncoghlan at gmail.com Sun Dec 1 23:17:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 08:17:48 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 Dec 2013 06:48, "Paul Moore" wrote: > > On 1 December 2013 19:21, Marcus Smith wrote: > >>> sometimes mean needing to build components with external dependencies > >>> from source > > > > you mean build once (or maybe after system updates for wheels with external > > binary deps), and cache as a local wheel, right? Right. The advantage of conda is that the use of hash based dependencies lets them distribute arbitrary binary dependencies, whether they're Python libraries or not. > Note that it is possible to convert binary packages from other formats > to wheel. We already support egg and bdist_wininst. If conda has built > packages that are worth supporting, I doubt that a conda-to-wheel > converter would be hard to write. I'd be willing to write one if > someone can point me at a spec for the conda package format (I > couldn't find one from a brief look through the docs). > > Conversely, if I have a wininst installer, a wheel or an egg, is there > a converter to conda format? I can't see having to use conda for some > things and pip for others as being a move towards lessening user > confusion. I see conda as existing at a similar level to apt and yum from a packaging point of view, with zc.buildout as a DIY equivalent at that level. For example, I installed Nikola into a virtualenv last night. That required installing the development headers for libxml2 and libxslt, but the error that tells you that is a C compiler one. I've been a C programmer longer than I have been a Python one, but I still had to resort to Google to try to figure out what dev libraries I needed. Outside the scientific space, crypto libraries are also notoriously hard to build, as are game engines and GUI toolkits. (I guess database bindings could also be a problem in some cases) We have the option to leave handling the arbitrary binary dependency problem to platforms, and I think we should take it. This is why I suspect there will be a better near term effort/reward trade-off in helping the conda folks improve the usability of their platform than there is in trying to expand the wheel format to cover arbitrary binary dependencies. Longer term, I expect we can expand pip's parallel install capabilities to support a better multi-version experience (after Python 3.4 is out the door, I intend to devote some quality time to trying to improve on the existing pkg_resources multi-version API now that I've been using it long enough to understand it's strengths and limitations) and potentially come up with a scheme for publishing supporting binaries inside prebuilt wheel files, but there are plenty of problems ahead of those on the todo list. By contrast, conda exists now and is being used happily by many members of the scientific and data analysis communities. It's not vapourware - it already works for its original target audience, and they love it. > I see conda (and enthought) as more like distributions - they sit in > the same space as rpms and debs. I don't see them as alternatives to > pip. Am I wrong? After all, both conda and enthought supply Python > interpreter builds as well as package repositories. And both feel like > unified ecosystems (you manage everything Python-related with conda) > rather than tools like pip (that works with existing Python standards > and interoperates with easy_install, distutils, etc). Exactly correct - in my view, the binary dependency management problem is exactly what defines the distinction between a cross-platform toolkit like pip and virtualenv and a new platform like conda. It's precisely the fact that conda defines a new platform that mostly ignores the underlying system that: - would make it a terrible choice for the core cross-platform software distribution toolkit - makes it much easier for them to consistently support a compiler free experience for end users Hence the layering proposal: if you're willing to build binary extensions from source occasionally, pip and virtualenv are already all you need. If you want someone else to handle the builds for you, rely on a platform instead. Like all platforms, conda has gaps in what it provides, but streamlining the PyPI to conda pipeline is going to be easier than streamlining inclusion in Linux distros, and Windows and Mac OS X have no equivalent binary dependency management system in the first place. > If the issue is simply around defining compatibility tags that better > describe the various environments around, then let's just get on with > that - we're going to have to do it in the end anyway, why temporarily > promote an alternative solution just to change our recommendation > later? That will get us to the point of correctly supporting self-contained extensions that only rely on the Python ABI and the platform C/C++ runtime, and I agree is a problem we need to solve at the cross-platform toolkit level. Conda is about solving the arbitrary binary dependency problem, in the same way other platforms do: by creating a pre-integrated collection of built libraries to cut down on the combinatorial explosion of possibilities. It isn't a coincidence that the "cross-platform platform" approach grew out of the scientific community - a solution like that is essential given the combination of ancient software with arcane build systems and end users that just want to run their data analysis rather than mess about with making the software work. My key point is that if you substitute "beginner" for "scientist" the desired end user experience is similar, but if you substitute "professional software developer" or "system integrator", it isn't - we often *want* (or need) to build from source and do our own integration, so a pre-integrated approach like conda is inappropriate to our use cases. I hadn't previously brought this up because I'm not entirely convinced conda is mature enough yet either, but as noted in the original post, it's become clear to me lately that people are *already* confused about how conda relates to pip, and that the lack of a shared understanding of how they differ has been causing friction on both sides. That friction is unnecessary - conda is no more a potential replacement for pip and virtualenv than zc.buildout is, but like zc.buildout, imposing additional constraints allows conda to solve problems that are difficult, perhaps even impossible, to handle at the core toolkit level. This also came up when Donald asked people to tell him about the problems they saw in packaging: Different tools: https://github.com/pypa/packaging-problems/issues/23 Build problems with standard tools: https://github.com/pypa/packaging-problems/issues/35 By being clear that we understand *why* the scientific community found a need to create their own tools, we can also start to make it clearer why their solutions to their specific problems don't extend to the full spectrum of use cases that need to be supported at the cross-platform toolkit level. Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Dec 1 23:20:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 08:20:42 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <1385910242.34561.YahooMailBasic@web171404.mail.ir2.yahoo.com> References: <1385910242.34561.YahooMailBasic@web171404.mail.ir2.yahoo.com> Message-ID: , On 2 Dec 2013 01:04, "Vinay Sajip" wrote: > > On Sun, 1/12/13, Nick Coghlan wrote: > > > (pyvenv doesn't offer an --always-copy option, just the option to use symlinks on > > It does - you should be able to run pyvenv with --copies to force copying, even on POSIX. Hmm, I didn't see that in --help. I'll take another look. Cheers, Nick. > > Regards, > > Vinay Sajip > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Dec 2 00:38:41 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 1 Dec 2013 23:38:41 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 1 December 2013 22:17, Nick Coghlan wrote: > For example, I installed Nikola into a virtualenv last night. That required > installing the development headers for libxml2 and libxslt, but the error > that tells you that is a C compiler one. > > I've been a C programmer longer than I have been a Python one, but I still > had to resort to Google to try to figure out what dev libraries I needed. But that's a *build* issue, surely? How does that relate to installing Nikola from a set of binary wheels? I understand you are thinking about non-Python libraries, but all I can say is that this has *never* been an issue to my knowledge in the Windows world. People either ship DLLs with the Python extension, or build statically. I understand that things are different in the Unix world, but to be blunt why should Windows users care? > Outside the scientific space, crypto libraries are also notoriously hard to > build, as are game engines and GUI toolkits. (I guess database bindings > could also be a problem in some cases) Build issues again... > We have the option to leave handling the arbitrary binary dependency problem > to platforms, and I think we should take it. Again, can we please be clear here? On Windows, there is no issue that I am aware of. Wheels solve the binary distribution issue fine in that environment (I know this is true, I've been using wheels for months now - sure there may be specialist areas that need some further work because they haven't had as much use yet, but that's details) > This is why I suspect there will be a better near term effort/reward > trade-off in helping the conda folks improve the usability of their platform > than there is in trying to expand the wheel format to cover arbitrary binary > dependencies. Excuse me if I'm feeling a bit negative towards this announcement. I've spent many months working on, and promoting, the wheel + pip solution, to the point where it is now part of Python 3.4. And now you're saying that you expect us to abandon that effort and work on conda instead? I never saw wheel as a pure-Python solution, installs from source were fine for me in that area. The only reason I worked so hard on wheel was to solve the Windows binary distribution issue. If the new message is that people should not distribute wheels for (for example) lxml, pyyaml, pymzq, numpy, scipy, pandas, gmpy, and pyside (to name a few that I use in wheel format relatively often) then effectively the work I've put in has been wasted. I'm hoping I've misunderstood here. Please clarify. Preferably with specifics for Windows (as "conda is a known stable platform" simply isn't true for me...) - I accept you're not a Windows user, so a pointer to already-existing documentation is fine (I couldn't find any myself). Paul. From qwcode at gmail.com Mon Dec 2 04:09:02 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sun, 1 Dec 2013 19:09:02 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > 2. For cross-platform handling of external binary dependencies, we > recommend boostrapping the open source conda toolchain, and using that > to install pre-built binaries (currently administered by the Continuum > Analytics folks). We already have a recommendation for conda: "If you?re looking for management of fully integrated cross-platform software stacks". That's admittedly pretty abstract. I certainly see no harm in expanding this so that it does a better job of attracting users that would seriously benefit. Maybe even link to a section on "Conda Environments" on the Advanced page. I don't have a good sense for the scope of what's currently available in anaconda to know how many users it could help. I guess I'd have to see the wording and placement of the recommendation, to judge it fully. My concern is just to prevent a horde of users trying to use conda environments that don't really need it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 2 08:31:09 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 17:31:09 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 09:38, Paul Moore wrote: > On 1 December 2013 22:17, Nick Coghlan wrote: > >> For example, I installed Nikola into a virtualenv last night. That required >> installing the development headers for libxml2 and libxslt, but the error >> that tells you that is a C compiler one. >> >> I've been a C programmer longer than I have been a Python one, but I still >> had to resort to Google to try to figure out what dev libraries I needed. > > But that's a *build* issue, surely? How does that relate to installing > Nikola from a set of binary wheels? Because libxml2 and libxslt aren't Python programs - they're external shared libraries. You would have to build statically for a wheel to work (which is, to be fair, exactly what we do in many cases - CPython itself bundles all its dependencies on Windows) > I understand you are thinking about non-Python libraries, but all I > can say is that this has *never* been an issue to my knowledge in the > Windows world. People either ship DLLs with the Python extension, or > build statically. I understand that things are different in the Unix > world, but to be blunt why should Windows users care? > >> Outside the scientific space, crypto libraries are also notoriously hard to >> build, as are game engines and GUI toolkits. (I guess database bindings >> could also be a problem in some cases) > > Build issues again... Yes, that's the point. wheels can solve the problem for cases where all external dependencies can be statically linked, or otherwise bundled inside the wheel. They *don't* work if you need multiple Python projects binding to the *same* copy of the external dependency. >> We have the option to leave handling the arbitrary binary dependency problem >> to platforms, and I think we should take it. > > Again, can we please be clear here? On Windows, there is no issue that > I am aware of. Wheels solve the binary distribution issue fine in that > environment (I know this is true, I've been using wheels for months > now - sure there may be specialist areas that need some further work > because they haven't had as much use yet, but that's details) Wheels work fine if they're self contained (whether through static linking or bundling), or only depending on other projects within the PyPI ecosystem, or if you're building custom wheels from source that only need to work in an environment you control. They *don't* work as soon as multiple components need to share a common external binary dependency that isn't part of the PyPI ecosystem, and you want to share your wheels with someone else - at that point you need to have a mechanism for providing the external dependencies as well. It's that second problem that some members of the scientific community have solved through conda, and that's the one I am saying we should postpone trying to solve at the pip level indefinitely (because it's damn hard, and we don't need to - the people that care about that feature generally won't care about the system integration features that pip offers over conda). >> This is why I suspect there will be a better near term effort/reward >> trade-off in helping the conda folks improve the usability of their platform >> than there is in trying to expand the wheel format to cover arbitrary binary >> dependencies. > > Excuse me if I'm feeling a bit negative towards this announcement. > I've spent many months working on, and promoting, the wheel + pip > solution, to the point where it is now part of Python 3.4. And now > you're saying that you expect us to abandon that effort and work on > conda instead? No, conda doesn't work for most of our use cases - it only works for the "I want this pre-integrated stack of software on this system, and I don't care about the details" use case. However, this is an area where pip/virtualenv can fall short because of the external shared binary dependency problem, and because there is a lot of bad software out there with appallingly unreliable build systems. If things are statically linked or bundled instead, then they fall into the scope that I agree pip/virtualenv *should* be able to handle (i.e. wheels that are self-contained assigned from the dependencies declared in their metadata). It's just the "arbitrary external shared binary dependency" problem that I want to put into the "will not solve" bucket. What that means is that remotely built wheels would *not* try to use system libraries, they would always use static linking and/or bundling for external binary dependencies. If you're already doing that on Windows, then *great*, that's working sensibly within the constraints of the design. Anyone that wanted dynamic linking of external shared dependencies would then need to either build from source (for integration with their local environment), or use a third party pre-integrated stack (like conda). > I never saw wheel as a pure-Python solution, installs > from source were fine for me in that area. The only reason I worked so > hard on wheel was to solve the Windows binary distribution issue. If > the new message is that people should not distribute wheels for (for > example) lxml, pyyaml, pymzq, numpy, scipy, pandas, gmpy, and pyside > (to name a few that I use in wheel format relatively often) then > effectively the work I've put in has been wasted. If a distribution can be sensibly published as a self-contained wheel with no external binary dependencies, then great, it makes sense to do that (e.g. an lxml wheel containing or statically linked to libxml2 and libxslt). The only problem I want to take off the table is the one where multiple wheel files try to share a dynamically linked external binary dependency. > I'm hoping I've misunderstood here. Please clarify. Preferably with > specifics for Windows (as "conda is a known stable platform" simply > isn't true for me...) - I accept you're not a Windows user, so a > pointer to already-existing documentation is fine (I couldn't find any > myself). Windows already has a culture of bundling all its dependencies, so what I'm suggesting isn't the least bit radical there. It *is* radical in other environments, as is the suggestion that the bundling philosophy of the CPython Windows installer be extended to all Windows targeted wheel files (at least for folks that mostly work on Linux). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Mon Dec 2 10:19:00 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 09:19:00 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 07:31, Nick Coghlan wrote: > The only problem I want to take off the table is the one where > multiple wheel files try to share a dynamically linked external binary > dependency. OK. Thanks for the clarification. Can I suggest that we need to be very careful how any recommendation in this area is stated? I certainly didn't get that impression from your initial posting, and from the other responses it doesn't look like I was the only one. We're only just starting to get real credibility for wheel as a distribution format, and we need to get a very strong message out that wheel is the future, and people should be distributing wheels as their primary binary format. My personal litmus test is the scientific community - when Christoph Gohlke is distributing his (Windows) binary builds as wheels, and projects like numpy, ipython, scipy etc are distributing wheels on PyPI, rather than bdist_wininst, I'll feel like we have got to the point where wheels are "the norm". The problem is, of course, that with conda being a scientific distribution at heart, any message we issue that promotes conda in any context will risk confusion in that community. My personal interest is as a non-scientific user who does a lot of data analysis, and finds IPython, Pandas, matplotlib numpy etc useful. At the moment I can pip install the tools I need (with a quick wheel convert from wininst format). I don't want to find that in the future I can't do that, but instead have to build from source or learn a new tool (conda). Paul From oscar.j.benjamin at gmail.com Mon Dec 2 11:45:30 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 2 Dec 2013 10:45:30 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 09:19, Paul Moore wrote: > On 2 December 2013 07:31, Nick Coghlan wrote: >> The only problem I want to take off the table is the one where >> multiple wheel files try to share a dynamically linked external binary >> dependency. > > OK. Thanks for the clarification. > > Can I suggest that we need to be very careful how any recommendation > in this area is stated? I certainly didn't get that impression from > your initial posting, and from the other responses it doesn't look > like I was the only one. I understood what Nick meant but I still don't understand how he's come to this conclusion. > We're only just starting to get real credibility for wheel as a > distribution format, and we need to get a very strong message out that > wheel is the future, and people should be distributing wheels as their > primary binary format. My personal litmus test is the scientific > community - when Christoph Gohlke is distributing his (Windows) binary > builds as wheels, and projects like numpy, ipython, scipy etc are > distributing wheels on PyPI, rather than bdist_wininst, I'll feel like > we have got to the point where wheels are "the norm". The problem is, > of course, that with conda being a scientific distribution at heart, > any message we issue that promotes conda in any context will risk > confusion in that community. Nick's proposal is basically incompatible with allowing Cristoph Gohlke to use pip and wheels. Christoph provides a bewildering array of installers for prebuilt packages that are interchangeable with other builds at the level of Python code but not necessarily at the binary level. So, for example, His scipy is incompatible with the "official" (from SourceForge) Windows numpy build because it links with the non-free Intel MKL library and it needs numpy to link against the same. Installing his scipy over the other numpy results in this: https://mail.python.org/pipermail//python-list/2013-September/655669.html So Christoph can provide wheels and people can manually download them and install from them but would beginners find that any easier than running the .exe installers? The .exe installers are more powerful and can do things like the numpy super-pack that distributes binaries for different levels of SSE support (as discussed previously on this list the wheel format cannot currently achieve this). Beginners will also find .exe installers more intuitive than running pip on the command line and will typically get better error messages etc. than pip provides. So I don't really see why Cristoph should bother switching formats (as noted by Paul before anyone who wants a wheel cache can easily convert his installers into wheels). AFAICT what Nick is saying is that it's not possible for pip and PyPI to guarantee the compatibility of different binaries because unlike apt-get and friends only part of the software stack is controlled. However I think this is not the most relevant difference between pip and apt-get here. The crucial difference is that apt-get communicates with repositories where all code and all binaries are under control of a single organisation. Pip (when used normally) communicates with PyPI and no single organisation controls the content of PyPI. So there's no way for pip/PyPI to guarantee *anything* about the compatibility of the code that they distribute/install, whether the problems are to do with binary compatibility or just compatibility of pure Python code. For pure Python distributions package authors are expected to solve the compatibility problems and pip provides version specifiers etc that they can use to do this. For built distributions they could do the same - except that pip/PyPI don't provide a mechanism for them to do so. Because PyPI is not a centrally controlled single software stack it needs a different model for ensuring compatibility - one driven by the community. People in the Python community are prepared to spend a considerable amount of time, effort and other resources solving this problem. Consider how much time Cristoph Gohlke must spend maintaining such a large internally consistent set of built packages. He has created a single compatible binary software stack for scientific computation. It's just that PyPI doesn't give him any way to distribute it. If perhaps he could own a tag like "cgohlke" and upload numpy:cgohlke and scipy:cgohlke then his scipy:cgohlke wheel could depend on numpy:cgohlke and numpy:cgohlke could somehow communicate the fact that it is incompatible with any other scipy distribution. This is one way in which pip/PyPI could facilitate the Python community to solve the binary compatibility problems. [As an aside I don't know whether Cristoph's Intel license would permit distribution via PYPI.] Another way would be to allow the community to create compatibility tags so that projects like numpy would have mechanisms to indicate e.g. Fortran ABI compatibility. In this model no one owns a particular tag but projects that depend on one another could simply use them in a consistent way that pip could understand. The impression I got from Nick's initial post is that, having discovered that the compatibility tags used in the wheel format are insufficient for the needs of the Python community and that it's not possible to enumerate the tags needed, pip/PyPI should just give up on the problem of binary compatibility. I think it would be better to think about simple mechanisms that the authors of the concerned packages could use so that people in the Python community can solve these problems for each of the packages they contribute to. There is enough will out there to make this work for all the big packages and problematic operating systems if only PyPI will allow it. Oscar From p.f.moore at gmail.com Mon Dec 2 12:57:13 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 11:57:13 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 10:45, Oscar Benjamin wrote: > Nick's proposal is basically incompatible with allowing Cristoph > Gohlke to use pip and wheels. Christoph provides a bewildering array > of installers for prebuilt packages that are interchangeable with > other builds at the level of Python code but not necessarily at the > binary level. So, for example, His scipy is incompatible with the > "official" (from SourceForge) Windows numpy build because it links > with the non-free Intel MKL library and it needs numpy to link against > the same. Installing his scipy over the other numpy results in this: > https://mail.python.org/pipermail//python-list/2013-September/655669.html Ah, OK. I had not seen this issue as I've always either used Christoph's builds or not used them. I've never tried or needed to mix builds. This is probably because I'm very much only a casual user of the scientific stack, so my needs are pretty simple. > So Christoph can provide wheels and people can manually download them > and install from them but would beginners find that any easier than > running the .exe installers? The .exe installers are more powerful and > can do things like the numpy super-pack that distributes binaries for > different levels of SSE support (as discussed previously on this list > the wheel format cannot currently achieve this). Beginners will also > find .exe installers more intuitive than running pip on the command > line and will typically get better error messages etc. than pip > provides. So I don't really see why Cristoph should bother switching > formats (as noted by Paul before anyone who wants a wheel cache can > easily convert his installers into wheels). The crucial answer here is that exe installers don't recognise virtualenvs. Again, I can imagine that a scientific user would naturally install Python and put all the scientific modules into the system Python - but precisely because I'm a casual user, I want to keep big dependencies like numpy/scipy out of my system Python, and so I use virtualenvs. The big improvement pip/wheel give over wininst is a consistent user experience, whether installing into the system Python, a virtualenv, or a Python 3.3+ venv. (I used to use wininsts in preference to pip, so please excuse a certain level of the enthusiasm of a convert here :-)) > AFAICT what Nick is saying is that it's not possible for pip and PyPI > to guarantee the compatibility of different binaries because unlike > apt-get and friends only part of the software stack is controlled. > However I think this is not the most relevant difference between pip > and apt-get here. The crucial difference is that apt-get communicates > with repositories where all code and all binaries are under control of > a single organisation. Pip (when used normally) communicates with PyPI > and no single organisation controls the content of PyPI. So there's no > way for pip/PyPI to guarantee *anything* about the compatibility of > the code that they distribute/install, whether the problems are to do > with binary compatibility or just compatibility of pure Python code. > For pure Python distributions package authors are expected to solve > the compatibility problems and pip provides version specifiers etc > that they can use to do this. For built distributions they could do > the same - except that pip/PyPI don't provide a mechanism for them to > do so. Agreed. Expecting the same level of compatibility guarantees from PyPI as is provided by RPM/apt is unrealistic, in my view. Heck, even pure Python packages don't give any indication as to whether they are Python 3 compatible in some cases (I just hit this today with the binstar package, as an example). This is a fact of life with a repository that doesn't QA uploads. > Because PyPI is not a centrally controlled single software stack it > needs a different model for ensuring compatibility - one driven by the > community. People in the Python community are prepared to spend a > considerable amount of time, effort and other resources solving this > problem. Consider how much time Cristoph Gohlke must spend maintaining > such a large internally consistent set of built packages. He has > created a single compatible binary software stack for scientific > computation. It's just that PyPI doesn't give him any way to > distribute it. If perhaps he could own a tag like "cgohlke" and upload > numpy:cgohlke and scipy:cgohlke then his scipy:cgohlke wheel could > depend on numpy:cgohlke and numpy:cgohlke could somehow communicate > the fact that it is incompatible with any other scipy distribution. > This is one way in which pip/PyPI could facilitate the Python > community to solve the binary compatibility problems. Exactly. > [As an aside I don't know whether Cristoph's Intel license would > permit distribution via PYPI.] Yes, I'd expect Cristoph's packages would likely always have to remain off PyPI (if for no other reason than the fact that he isn't the owner of the packages he's providing distributions for). But if he did provide a PyPI style index, compatibility could be handled manually via `pip install --index-url `. And even if they are not on PyPI, your custom tag suggestion above would still be beneficial. > Another way would be to allow the community to create compatibility > tags so that projects like numpy would have mechanisms to indicate > e.g. Fortran ABI compatibility. In this model no one owns a particular > tag but projects that depend on one another could simply use them in a > consistent way that pip could understand. > > The impression I got from Nick's initial post is that, having > discovered that the compatibility tags used in the wheel format are > insufficient for the needs of the Python community and that it's not > possible to enumerate the tags needed, pip/PyPI should just give up on > the problem of binary compatibility. I think it would be better to > think about simple mechanisms that the authors of the concerned > packages could use so that people in the Python community can solve > these problems for each of the packages they contribute to. There is > enough will out there to make this work for all the big packages and > problematic operating systems if only PyPI will allow it. Agreed - completely giving up on the issue in favour of a separately curated solution seems wrong to me, at least in the sense that it abandons people who don't fit cleanly into either solution (e.g., someone who needs to use virtualenv rather than conda environments, but needs numpy, and a package that Christoph distributes, but conda doesn't - who should that person turn to for help?). I don't see any problem with admitting we can't solve every aspect of the problem automatically, but that doesn't preclude providing extensibility to allow communities to solve the parts of the issue that impact their own area. As a quick sanity check question - what is the long-term advice for Christoph (and others like him)? Continue distributing wininst installers? Move to wheels? Move to conda packages? Do whatever you want, we don't care? We're supposedly pushing pip as "the officially supported solution to package management" - how can that be reconciled with *not* advising builders[1] to produce pip-compatible packages? Paul [1] At least, those who are not specifically affiliated with a group offering a curated solution (like conda or ActiveState). From ncoghlan at gmail.com Mon Dec 2 14:22:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 2 Dec 2013 23:22:21 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 Dec 2013 21:57, "Paul Moore" wrote: > > On 2 December 2013 10:45, Oscar Benjamin wrote: > > Nick's proposal is basically incompatible with allowing Cristoph > > Gohlke to use pip and wheels. Christoph provides a bewildering array > > of installers for prebuilt packages that are interchangeable with > > other builds at the level of Python code but not necessarily at the > > binary level. So, for example, His scipy is incompatible with the > > "official" (from SourceForge) Windows numpy build because it links > > with the non-free Intel MKL library and it needs numpy to link against > > the same. Installing his scipy over the other numpy results in this: > > https://mail.python.org/pipermail//python-list/2013-September/655669.html > > Ah, OK. I had not seen this issue as I've always either used > Christoph's builds or not used them. I've never tried or needed to mix > builds. This is probably because I'm very much only a casual user of > the scientific stack, so my needs are pretty simple. > > > So Christoph can provide wheels and people can manually download them > > and install from them but would beginners find that any easier than > > running the .exe installers? The .exe installers are more powerful and > > can do things like the numpy super-pack that distributes binaries for > > different levels of SSE support (as discussed previously on this list > > the wheel format cannot currently achieve this). Beginners will also > > find .exe installers more intuitive than running pip on the command > > line and will typically get better error messages etc. than pip > > provides. So I don't really see why Cristoph should bother switching > > formats (as noted by Paul before anyone who wants a wheel cache can > > easily convert his installers into wheels). > > The crucial answer here is that exe installers don't recognise > virtualenvs. Again, I can imagine that a scientific user would > naturally install Python and put all the scientific modules into the > system Python - but precisely because I'm a casual user, I want to > keep big dependencies like numpy/scipy out of my system Python, and so > I use virtualenvs. > > The big improvement pip/wheel give over wininst is a consistent user > experience, whether installing into the system Python, a virtualenv, > or a Python 3.3+ venv. (I used to use wininsts in preference to pip, > so please excuse a certain level of the enthusiasm of a convert here > :-)) And the conda folks are working on playing nice with virtualenv - I don't we'll see a similar offer from Microsoft for MSI any time soon :) > > AFAICT what Nick is saying is that it's not possible for pip and PyPI > > to guarantee the compatibility of different binaries because unlike > > apt-get and friends only part of the software stack is controlled. > > However I think this is not the most relevant difference between pip > > and apt-get here. The crucial difference is that apt-get communicates > > with repositories where all code and all binaries are under control of > > a single organisation. Pip (when used normally) communicates with PyPI > > and no single organisation controls the content of PyPI. So there's no > > way for pip/PyPI to guarantee *anything* about the compatibility of > > the code that they distribute/install, whether the problems are to do > > with binary compatibility or just compatibility of pure Python code. > > For pure Python distributions package authors are expected to solve > > the compatibility problems and pip provides version specifiers etc > > that they can use to do this. For built distributions they could do > > the same - except that pip/PyPI don't provide a mechanism for them to > > do so. > > Agreed. Expecting the same level of compatibility guarantees from PyPI > as is provided by RPM/apt is unrealistic, in my view. Heck, even pure > Python packages don't give any indication as to whether they are > Python 3 compatible in some cases (I just hit this today with the > binstar package, as an example). This is a fact of life with a > repository that doesn't QA uploads. Exactly, this is the difference between pip and conda - conda is a solution for installing from curated *collections* of packages. It's somewhat related to the tagging system people are speculating about for PyPI, but instead of being purely hypothetical, it already exists. Because it uses hash based dependencies, there's no chance of things getting mixed up. That design has other problems which limit the niche where a tool like conda is the right answer, but within that niche, hash based dependency management helps bring the combinatorial explosion of possible variations under control. > > Because PyPI is not a centrally controlled single software stack it > > needs a different model for ensuring compatibility - one driven by the > > community. People in the Python community are prepared to spend a > > considerable amount of time, effort and other resources solving this > > problem. Consider how much time Cristoph Gohlke must spend maintaining > > such a large internally consistent set of built packages. He has > > created a single compatible binary software stack for scientific > > computation. It's just that PyPI doesn't give him any way to > > distribute it. If perhaps he could own a tag like "cgohlke" and upload > > numpy:cgohlke and scipy:cgohlke then his scipy:cgohlke wheel could > > depend on numpy:cgohlke and numpy:cgohlke could somehow communicate > > the fact that it is incompatible with any other scipy distribution. > > This is one way in which pip/PyPI could facilitate the Python > > community to solve the binary compatibility problems. > > Exactly. > > > [As an aside I don't know whether Cristoph's Intel license would > > permit distribution via PYPI.] > > Yes, I'd expect Cristoph's packages would likely always have to remain > off PyPI (if for no other reason than the fact that he isn't the owner > of the packages he's providing distributions for). But if he did > provide a PyPI style index, compatibility could be handled manually > via `pip install --index-url `. And even if they are > not on PyPI, your custom tag suggestion above would still be > beneficial. Yes, I think a "variant" tag would be a useful feature for wheels as well. It's not a substitute for hash identified fully curated stacks, though. > > > Another way would be to allow the community to create compatibility > > tags so that projects like numpy would have mechanisms to indicate > > e.g. Fortran ABI compatibility. In this model no one owns a particular > > tag but projects that depend on one another could simply use them in a > > consistent way that pip could understand. > > > > The impression I got from Nick's initial post is that, having > > discovered that the compatibility tags used in the wheel format are > > insufficient for the needs of the Python community and that it's not > > possible to enumerate the tags needed, pip/PyPI should just give up on > > the problem of binary compatibility. I think it would be better to > > think about simple mechanisms that the authors of the concerned > > packages could use so that people in the Python community can solve > > these problems for each of the packages they contribute to. There is > > enough will out there to make this work for all the big packages and > > problematic operating systems if only PyPI will allow it. > > Agreed - completely giving up on the issue in favour of a separately > curated solution seems wrong to me, at least in the sense that it > abandons people who don't fit cleanly into either solution (e.g., > someone who needs to use virtualenv rather than conda environments, > but needs numpy, and a package that Christoph distributes, but conda > doesn't - who should that person turn to for help?). I don't see any > problem with admitting we can't solve every aspect of the problem > automatically, but that doesn't preclude providing extensibility to > allow communities to solve the parts of the issue that impact their > own area. We already have more than enough work to do in the packaging space - why come up with a new solution for publication of curated stacks when the conda folks already have one, and it works cross-platform for the stack with the most complex external dependencies? If static linking or bundling is an option, then wheels can already handle it. If targeting a particular controlled environment (or a user base prepared to get the appropriate bits in place themselves), wheels can also handle shared external binary dependencies. What they can't easily do is share a fully curated stack of software, including external binary dependencies, in a way that works across platforms and isn't trivially easy to screw up by accidentally installing from the wrong place. Just as a while ago I started thinking it made more sense to rehabilitate setuptools than it did to try to replace it any time soon, and we mostly let the TUF folks run with the end-to-end security problem, I now think it makes sense to start working on collaborating with the conda folks on the "distribution of curated software stacks" problem, as that reduces the pressure on the core tools to address that concern. PyPI wheels would then be about publishing "default" versions of components, with the broadest compatibility, while conda would be a solution for getting access to alternate builds that may be faster, but require external shared dependencies. > As a quick sanity check question - what is the long-term advice for > Christoph (and others like him)? Continue distributing wininst > installers? Move to wheels? Move to conda packages? Do whatever you > want, we don't care? We're supposedly pushing pip as "the officially > supported solution to package management" - how can that be reconciled > with *not* advising builders[1] to produce pip-compatible packages? What Christoph is doing is producing a cross-platform curated binary software stack, including external dependencies. That's precisely the problem I'm suggesting we *not* try to solve in the core tools any time soon, but instead support bootstrapping conda to solve the problem at a different layer. So the pip compatible builds for those tools would likely miss out on some of the external acceleration features, while curated stacks built with other options could be made available through conda (using hash based dependencies to ensure consistency). Revising the wheel spec to include variant tags, or improve the way platforms are identified is a long way down the todo list, and that's without allowing the subsequent time needed update pip and other tools. By ceding the "distribution of cross-platform curated software stacks with external binary dependencies" problem to conda, users would get a solution to that problem that they can use *now*, rather than in some indefinite future after the metadata 2.0 and end-to-end security changes in the core tools have been resolved. Now, it may be we take a closer look at conda and decide there are critical issues that need to be addressed before it can be recommended rather than just mentioned (e.g. I don't know off the top of my head if the server comms is appropriately secured). This thread was mostly about pointing that there's a thorny subset of the cross-platform software distribution problem that I think we can offload to someone else and have something that mostly works *today*, rather than only getting to it in some speculative future after a bunch of additional currently hand-wavey design work has been completed. Cheers, Nick. > > Paul > > [1] At least, those who are not specifically affiliated with a group > offering a curated solution (like conda or ActiveState). -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Mon Dec 2 14:29:32 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 02 Dec 2013 08:29:32 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <1385935657.29586.YahooMailBasic@web171401.mail.ir2.yahoo.com> References: <1385935657.29586.YahooMailBasic@web171401.mail.ir2.yahoo.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/01/2013 05:07 PM, Vinay Sajip wrote: > On Sun, 1/12/13, Paul Moore wrote: > >> If the issue is simply around defining compatibility tags that >> better describe the various environments around, then let's just >> get on with that - we're going to have to do it in the end anyway, >> why temporarily promote an alternative solution just to change our >> recommendation later? > > This makes sense to me. We should refine the compatibility tags as > much as is required. It would be nice if there was some place (on > PyPI, or elsewhere) where users could request binary distributions for > specific packages for particular environments, and then some kind > people with those environments might be able to build those wheels and > upload them ... a bit like Christoph Gohlke does for Windows. The issue is combinatorial explosion in the compatibility tag space. There is basically zero chance that even Linux users (even RedHat users across RHEL version) would benefit from pre-built binary wheels (as opposed to packages from their distribution). Wheels on POSIX allow caching of the build process for deployment across a known set of hosts: they won't insulate you from the need to build in the first place. Wheels *might* be in play in the for-pay market, where a vendor supports a limited set platforms, but those solutions will use separate indexes anyway. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKcizYACgkQ+gerLs4ltQ6kKwCfRa5s8XnM5SwlnnIHGGJ8dJSg hPUAn1TLWQNxtbQmPvvMPT2rEmlhCwq5 =xRsn -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Dec 2 14:34:59 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 02 Dec 2013 08:34:59 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/01/2013 05:17 PM, Nick Coghlan wrote: > I see conda as existing at a similar level to apt and yum from a > packaging point of view, with zc.buildout as a DIY equivalent at that > level. FTR: zc.buildout does nothing to insulate you from the need for a compiler; it does allow you to create repeatable builds from source for non-Python components which would otherwise vary with the underlying platform. The actual recipes for such components often involve a *lot* of yak shaving. ;) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKcjIMACgkQ+gerLs4ltQ5XlQCeMmoyvAOvJGChhpGOF2Phkut0 nfwAnjj2pbr8bHKfS8+lzt/XorPVNzSe =QmuK -----END PGP SIGNATURE----- From tseaver at palladion.com Mon Dec 2 14:38:00 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 02 Dec 2013 08:38:00 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/01/2013 06:38 PM, Paul Moore wrote: > I understand that things are different in the Unix world, but to be > blunt why should Windows users care? You're kidding, right? 90% or more of the reason for wheels in the first place is because Windows users can't build their own software from source. The amount of effort put in by non-Windows package owners to support them dwarfs whatever is bothering you here. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKcjTgACgkQ+gerLs4ltQ7fQQCg0Pfd5tp3vvEsJnJ0aNLNeIXH bVwAn2av6wxVMXEqe4jIQLL+2W4oqQ9G =foOx -----END PGP SIGNATURE----- From ralf.gommers at gmail.com Mon Dec 2 08:20:08 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Mon, 2 Dec 2013 08:20:08 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Mon, Dec 2, 2013 at 12:38 AM, Paul Moore wrote: > On 1 December 2013 22:17, Nick Coghlan wrote: > > > For example, I installed Nikola into a virtualenv last night. That > required > > installing the development headers for libxml2 and libxslt, but the error > > that tells you that is a C compiler one. > > > > I've been a C programmer longer than I have been a Python one, but I > still > > had to resort to Google to try to figure out what dev libraries I needed. > > But that's a *build* issue, surely? How does that relate to installing > Nikola from a set of binary wheels? > > I understand you are thinking about non-Python libraries, but all I > can say is that this has *never* been an issue to my knowledge in the > Windows world. People either ship DLLs with the Python extension, or > build statically. I understand that things are different in the Unix > world, but to be blunt why should Windows users care? > > > Outside the scientific space, crypto libraries are also notoriously hard > to > > build, as are game engines and GUI toolkits. (I guess database bindings > > could also be a problem in some cases) > > Build issues again... > > > We have the option to leave handling the arbitrary binary dependency > problem > > to platforms, and I think we should take it. > > Again, can we please be clear here? On Windows, there is no issue that > I am aware of. Wheels solve the binary distribution issue fine in that > environment (I know this is true, I've been using wheels for months > now - sure there may be specialist areas that need some further work > because they haven't had as much use yet, but that's details) > > > This is why I suspect there will be a better near term effort/reward > > trade-off in helping the conda folks improve the usability of their > platform > > than there is in trying to expand the wheel format to cover arbitrary > binary > > dependencies. > > Excuse me if I'm feeling a bit negative towards this announcement. > I've spent many months working on, and promoting, the wheel + pip > solution, to the point where it is now part of Python 3.4. And now > you're saying that you expect us to abandon that effort and work on > conda instead? I never saw wheel as a pure-Python solution, installs > from source were fine for me in that area. The only reason I worked so > hard on wheel was to solve the Windows binary distribution issue. If > the new message is that people should not distribute wheels for (for > example) lxml, pyyaml, pymzq, numpy, scipy, pandas, gmpy, and pyside > (to name a few that I use in wheel format relatively often) then > effectively the work I've put in has been wasted. > Hi, scipy developer here. In the scientific python community people are definitely interested in and intending to standardize on wheels. Your work on wheel + pip is much appreciated. The problems above that you say are "build issues" aren't really build issues (where build means what distutils/bento do to build a package). Maybe the following concepts, shamelessly stolen from the thread linked below, help: - *build systems* handle the actual building of software, eg Make, CMake, distutils, Bento, autotools, etc - *package managers* handle the distribution and installation of built (or source) software, eg pip, apt, brew, ports - *build managers* are separate from the above and handle the automatic(?) preparation of packages from the results of build systems Conda is a package manager to the best of my understanding, but because it controls the whole stack it can also already do parts of the job of a build manager. This is not something that pip aims to do. Conda is fairly new and not well understood in our community either, but maybe this (long) thread helps: https://groups.google.com/forum/#!searchin/numfocus/build$20managers/numfocus/mVNakFqfpZg/6h_SldGNM-EJ. Regards, Ralf I'm hoping I've misunderstood here. Please clarify. Preferably with > specifics for Windows (as "conda is a known stable platform" simply > isn't true for me...) - I accept you're not a Windows user, so a > pointer to already-existing documentation is fine (I couldn't find any > myself). > > Paul. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From imran at smitsol.com Mon Dec 2 08:53:25 2013 From: imran at smitsol.com (Imran M Yousuf) Date: Mon, 2 Dec 2013 13:53:25 +0600 Subject: [Distutils] Package a project Message-ID: Hi, I am new to setuptools. I am using it to build and package a project of mine. Currently if I execute `python setup.py bdist` it generates a tarball with all files located in paths './abs/path/to/project/bin/[entry points]' and './abs/path/to/project/lib/python-2.7/site-packages/[rest of the sources]'. This does not seem to be logical :(, I would rather want the binary distribution to be structure - './project-name/bin/' and './project-name/lib/'. Can some please advise me how to achieve it? I am using VirtualEnv for development of this project and its setup.py looks like - from setuptools import setup, find_packages setup(name='project-name', version='1.0', description='Description', author='Imran M Yousuf', author_email='imran at smitsol.com', url='http://www.smitsol.com', install_requires = ['setuptools', 'pycrypto==2.6'], packages=find_packages('src', ["tests"]), package_dir={'': 'src'}, test_suite="tests", entry_points={ 'console_scripts': ['manager=client.manager:main'] } ) Thank you, -- Imran M Yousuf Entrepreneur & CEO Smart IT Solution http://smitsol.com 25/5B, Block F, Haji Chinu Miah Road Bylane Joint Quarter, Mohammadpur Dhaka - 1207, Bangladesh Email: imran at smitsol.com Twitter: @imyousuf - http://twitter.com/imyousuf Skype: imyousuf Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557 +880-1746119494 From p.f.moore at gmail.com Mon Dec 2 14:54:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 13:54:17 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 13:22, Nick Coghlan wrote: >> As a quick sanity check question - what is the long-term advice for >> Christoph (and others like him)? Continue distributing wininst >> installers? Move to wheels? Move to conda packages? Do whatever you >> want, we don't care? We're supposedly pushing pip as "the officially >> supported solution to package management" - how can that be reconciled >> with *not* advising builders[1] to produce pip-compatible packages? > > What Christoph is doing is producing a cross-platform curated binary > software stack, including external dependencies. That's precisely the > problem I'm suggesting we *not* try to solve in the core tools any time > soon, but instead support bootstrapping conda to solve the problem at a > different layer. OK. From my perspective, that's *not* what Christoph is doing (I concede that it might be from his perspective, though). As far as I know, the only place where Christoph's builds are incompatible with standard builds is where numpy is involved (where he uses Intel compiler extensions). But what he does *for me* is to provide binary builds of lxml, pyyaml, matplotlib, pyside and a number of other packages that I haven't got the infrastructure set up locally to build. [He also provides apparently-incompatible binary builds of scientific packages like numpy/scipy/pandas, but that's a side-issue and as I get *all* of my scientific packages from him, the incompatibility is not a visible problem for me] If the named projects provided Windows binaries, then there would be no issue with Christoph's stuff. But AFAIK, there is no place I can get binary builds of matplotlib *except* from Christoph. And lxml provides limited sets of binaries - there's no Python 3.3 version, for example. I could continue :-) Oh, and by the way, in what sense do you mean "cross-platform" here? Win32 and Win64? Maybe I'm being narrow minded, but I tend to view "cross platform" as meaning "needs to think about at least two of Unix, Windows and OSX". The *platform* issues on Windows (and OSX, I thought) are solved - it's the ABI issues that we've ignored thus far (successfully till now :-)) But Christoph's site won't go away because of this debate, and as long as I can find wininst, egg or wheel binaries somewhere, I can maintain my own personal wheel index. So I don't really care much, and I'll stop moaning for now. I'll focus my energies on building that personal index instead. Paul From p.f.moore at gmail.com Mon Dec 2 14:59:10 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 13:59:10 +0000 Subject: [Distutils] Package a project In-Reply-To: References: Message-ID: On 2 December 2013 07:53, Imran M Yousuf wrote: > Hi, > > I am new to setuptools. I am using it to build and package a project > of mine. Currently if I execute `python setup.py bdist` it generates a > tarball with all files located in paths > './abs/path/to/project/bin/[entry points]' and > './abs/path/to/project/lib/python-2.7/site-packages/[rest of the > sources]'. This does not seem to be logical :(, I would rather want > the binary distribution to be structure - > './project-name/bin/' and './project-name/lib/'. > > Can some please advise me how to achieve it? I am using VirtualEnv for > development of this project and its setup.py looks like - > > from setuptools import setup, find_packages > > setup(name='project-name', > version='1.0', > description='Description', > author='Imran M Yousuf', > author_email='imran at smitsol.com', > url='http://www.smitsol.com', > install_requires = ['setuptools', 'pycrypto==2.6'], > packages=find_packages('src', ["tests"]), > package_dir={'': 'src'}, > test_suite="tests", > entry_points={ > 'console_scripts': ['manager=client.manager:main'] > } > ) Install the wheel project and use bdist_wheel instead of a simple bdist. Also, use the sdist (source distribution) command to create a source package (that needs a compiler to build). Binary packages are only compatible with the platform/Python version they are built on, so you may want to make multiple wheels, depending on what platforms you are targeting. >From what you provide, I'm not 100% sure if you have C code in your project, actually. If you don't, then a sdist is sufficient - although a wheel might be worth uploading as well (pure Python wheels are cross-platform). The plain bdist command produces a "dumb" binary distribution, which is obsolete, and frankly useless. Paul From info at michael-jansen.biz Mon Dec 2 15:14:52 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Mon, 02 Dec 2013 15:14:52 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin Message-ID: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> Hi I am currently working on the cobbler (http://cobblerd.org) setup.py and trying to improve it. It currently is only capable of installing to /usr for several reasons. Since i would like to have virtualenv support when running it i am trying to change that. While doing that i managed to meet one of the standard problems with distutils/setuptools - a search returns results of 2004 or older - which still seems to be unsolved. I would be willing to change that but my recent research leaves me a bit confused with all that setuptools is not dead, distlibs, tuf and wheels stuff. So here is the question. If i am willing to do the work is it possible to add something like sbin_scripts (or admin_scripts?) to distutils? I locally solved the problem like that http://pastebin.kde.org/pqrwrud1p (or attached). The implementation expands INSTALL_SCHEMA and reuses install_scripts to install those sbin_scripts too. That could be debatable. Perhaps a dedicated build|install_xxx_scripts command. I want to add that fhs also has a libexec folder (usually /usr/lib//libexec these days) destined for scripts that are supposed to be only called by other scripts/programs not manually. And cobbler for installs python scripts for modwsgi into /srv/www/cobbler/ . So i guess just adding sbin_scripts is not really the solution. Perhaps something more flexible is needed. Or should i wait for nextgen distlib based swiss knife python build tool and for now keep my local additions (there are more). -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- from distutils.command import install as dist_install from distutils.command.build import build as _build from distutils.command.build_scripts import build_scripts as _build_scripts from distutils.command.build_scripts import convert_path, newer, first_line_re from setuptools.command.install import install as _install from setuptools.command.install_scripts import install_scripts as _install_scripts from setuptools import setup as _setup from setuptools.dist import Distribution as _Distribution from stat import ST_MODE import sys import os from distutils import log if sys.version < "2.2": dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' else: dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['unix_prefix']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_home']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_user']['sbin_scripts'] = '$userbase/sbin' dist_install.INSTALL_SCHEMES['nt_user']['sbin_scripts'] = '$userbase/Scripts' dist_install.INSTALL_SCHEMES['os2']['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['os2_home']['sbin_scripts'] = '$userbase/sbin' # The keys to an installation scheme; if any new types of files are to be # installed, be sure to add an entry to every installation scheme above, # and to SCHEME_KEYS here. dist_install.SCHEME_KEYS = ('purelib', 'platlib', 'headers', 'scripts', 'data') + ('sbin_scripts',) class install(_install): """Enhance install target aimed for inclusion upstream.""" user_options = _install.user_options + [ ('install-sbin-scripts=', None, "installation directory for sbin scripts") ] def initialize_options(self): _install.initialize_options(self) self.install_sbin_scripts = None def finalize_options(self): _install.finalize_options(self) if self.root is not None: self.change_roots('sbin_scripts') def finalize_unix(self): _install.finalize_unix(self) if self.install_base is not None or self.install_platbase is not None: if self.install_sbin_scripts is None: raise DistutilsOptionError, \ ("install-base or install-platbase supplied, but " "installation scheme is incomplete") return def finalize_other(self): _install.finalize_other(self) # Nothing else to do here def expand_dirs(self): _install.expand_dirs(self) self._expand_attrs(['install_sbin_scripts']) # :TODO: # In run() add it to rejectdirs class install_scripts(_install_scripts): user_options = _install_scripts.user_options + [ ('install-sbin-dir=', 'd', "directory to install sbin scripts to") ] def initialize_options(self): _install_scripts.initialize_options(self) self.install_sbin_dir = None self.build_sbin_dir = None def finalize_options (self): _install_scripts.finalize_options(self) self.set_undefined_options( 'build', ('build_sbin_scripts', 'build_sbin_dir')) self.set_undefined_options( 'install', ('install_sbin_scripts', 'install_sbin_dir') ) def run(self): _install_scripts.run(self) print self.build_sbin_dir print self.install_sbin_dir self.outfiles = self.copy_tree(self.build_sbin_dir, self.install_sbin_dir) if os.name == 'posix': # Set the executable bits (owner, group, and world) on # all the scripts we just installed. for file in self.get_outputs(): if self.dry_run: log.info("changing mode of %s", file) else: mode = ((os.stat(file)[ST_MODE]) | 0555) & 07777 log.info("changing mode of %s to %o", file, mode) os.chmod(file, mode) class build(_build): user_options = _build_scripts.user_options + [ ('build-sbin-scripts=', 'd', "directory to \"build\" (copy) to") ] def initialize_options(self): _build.initialize_options(self) self.build_sbin_scripts = None def finalize_options(self): _build.finalize_options(self) self.build_sbin_scripts = None if self.build_sbin_scripts is None: self.build_sbin_scripts = os.path.join(self.build_base, 'sbin_scripts-' + sys.version[0:3]) class build_scripts(_build_scripts): user_options = _build_scripts.user_options + [ ('build-sbin-dir=', 'd', "directory to \"build\" (copy) to") ] def initialize_options(self): _build_scripts.initialize_options(self) self.build_sbin_dir = None def finalize_options(self): _build_scripts.finalize_options(self) self.set_undefined_options( 'build', ('build_sbin_scripts', 'build_sbin_dir') ) self.sbin_scripts = self.distribution.sbin_scripts def copy_scripts (self): """Copy each script listed in 'self.scripts'; if it's marked as a Python script in the Unix way (first line matches 'first_line_re', ie. starts with "\#!" and contains "python"), then adjust the first line to refer to the current Python interpreter as we copy. """ _build_scripts.copy_scripts(self) print "HERE" _sysconfig = __import__('sysconfig') self.mkpath(self.build_sbin_dir) outfiles = [] for script in self.sbin_scripts: adjust = 0 script = convert_path(script) outfile = os.path.join(self.build_sbin_dir, os.path.basename(script)) outfiles.append(outfile) if not self.force and not newer(script, outfile): log.debug("not copying %s (up-to-date)", script) continue # Always open the file, but ignore failures in dry-run mode -- # that way, we'll get accurate feedback if we can read the # script. try: f = open(script, "r") except IOError: if not self.dry_run: raise f = None else: first_line = f.readline() if not first_line: self.warn("%s is an empty file (skipping)" % script) continue match = first_line_re.match(first_line) if match: adjust = 1 post_interp = match.group(1) or '' if adjust: log.info("copying and adjusting %s -> %s", script, self.build_dir) if not self.dry_run: outf = open(outfile, "w") if not _sysconfig.is_python_build(): outf.write("#!%s%s\n" % (self.executable, post_interp)) else: outf.write("#!%s%s\n" % (os.path.join( _sysconfig.get_config_var("BINDIR"), "python%s%s" % (_sysconfig.get_config_var("VERSION"), _sysconfig.get_config_var("EXE"))), post_interp)) outf.writelines(f.readlines()) outf.close() if f: f.close() else: if f: f.close() self.copy_file(script, outfile) if os.name == 'posix': for file in outfiles: if self.dry_run: log.info("changing mode of %s", file) else: oldmode = os.stat(file)[ST_MODE] & 07777 newmode = (oldmode | 0555) & 07777 if newmode != oldmode: log.info("changing mode of %s from %o to %o", file, oldmode, newmode) os.chmod(file, newmode) class Distribution(_Distribution): def __init__(self, *args, **kwargs): self.sbin_scripts = [] _Distribution.__init__(self, *args, **kwargs) def setup(*args, **kwargs): commands = { 'install': install, 'install_scripts': install_scripts, 'build': build, 'build_scripts': build_scripts } if 'cmdclass' in kwargs: for name in commands.keys(): if name not in kwargs['cmdclass']: kwargs['cmdclass'][name] = commands[name] _setup(*args, **kwargs) From p.f.moore at gmail.com Mon Dec 2 15:19:25 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 14:19:25 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 13:38, Tres Seaver wrote: > On 12/01/2013 06:38 PM, Paul Moore wrote: >> I understand that things are different in the Unix world, but to be >> blunt why should Windows users care? > > You're kidding, right? 90% or more of the reason for wheels in the first > place is because Windows users can't build their own software from > source. The amount of effort put in by non-Windows package owners to > support them dwarfs whatever is bothering you here. My point is that most of the complex binary compatibility problems seem to be Unix-related, and as you imply, Unix users don't seem to have much interest in using wheels except for local caching. So why build that complexity into the spec if the main users (Windows, and Unix users who won't ever publish wheels outside their own systems) don't have a need for it? Let's just stick with something simple that has limitations but works (practicality beats purity). My original bdist_simple proposal was a pure-Windows replacement for wininst. Daniel developed that into wheels which cater for non-Windows systems (I believe, precisely because he had an interest in the local cache use case). We're now seeing the complexities of the Unix world affect the design of wheels, and it's turning out to be a hard problem. All I'm trying to say is let's not give up on binary wheels for Windows, just because we have unsolved issues on Unix. Whether solving the Unix issues is worth it is the Unix users' call - I'll help solve the issues, if they choose to, but I won't support abandoning the existing Windows solution just because it can't be extended to cater for Unix as well. I'm immensely grateful for the amount of work projects which are developed on Unix (and 3rd parties like Cristoph) put into supporting Windows. Far from dismissing that, I want to avoid making things any harder than they already are for such people - current wheels are no more complex to distribute than wininst installers, and I want to keep the impact on non-Windows projects at that level. If I come across as ungrateful, I apologise. Paul From oscar.j.benjamin at gmail.com Mon Dec 2 15:19:43 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 2 Dec 2013 14:19:43 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 13:54, Paul Moore wrote: > > If the named projects provided Windows binaries, then there would be > no issue with Christoph's stuff. But AFAIK, there is no place I can > get binary builds of matplotlib *except* from Christoph. And lxml > provides limited sets of binaries - there's no Python 3.3 version, for > example. I could continue :-) The matplotlib folks provide a list of binaries for Windows and OSX hosted by SourceForge: http://matplotlib.org/downloads.html So do numpy and scipy. > Oh, and by the way, in what sense do you mean "cross-platform" here? > Win32 and Win64? Maybe I'm being narrow minded, but I tend to view > "cross platform" as meaning "needs to think about at least two of > Unix, Windows and OSX". The *platform* issues on Windows (and OSX, I > thought) are solved - it's the ABI issues that we've ignored thus far > (successfully till now :-)) Exactly. A python extension that uses Fortran needs to indicate which of the two Fortran ABIs it uses. Scipy must use the same ABI as the BLAS/LAPACK library that numpy was linked with. This is core compatibility data but there's no way to communicate it to pip. There's no need to actually provide downloadable binaries for both ABIs but there is a need to be able to detect incompatibilities. Basically if 1) There is at least one single consistent set of built wheels for Windows/OSX for any popular set of binary-interdependent packages. 2) A way to automatically detect incompatibilities and to automatically find compatible built wheels. then *a lot* of packaging problems have been solved. Part 1 already exists. There are multiple consistent sets of built installers (not wheels yet) for many hard to build packages. Part 2 requires at least some changes in pip/PyPI. I read somewhere that numpy is the most frequently cited dependency on PyPI. It can be built in multiple binary-incompatible ways. If there is at least a way for the installer to know that it was built in "the standard way" (for Windows/OSX) then there can be a set of binaries built to match that. There's no need for a combinatorial explosion of compatibility tags - just a single set of compatibility tags that has complete binaries (where the definition of complete obviously depends on your field). People who want to build in different incompatible ways can do so themselves, although it would still be nice to get an install time error message when you subsequently try to install something incompatible. For Linux this problem is basically solved as far as beginners are concerned because they can just use apt. Oscar From p.f.moore at gmail.com Mon Dec 2 16:28:20 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 15:28:20 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 14:19, Oscar Benjamin wrote: > Basically if > 1) There is at least one single consistent set of built wheels for > Windows/OSX for any popular set of binary-interdependent packages. > 2) A way to automatically detect incompatibilities and to > automatically find compatible built wheels. > then *a lot* of packaging problems have been solved. > > Part 1 already exists. There are multiple consistent sets of built > installers (not wheels yet) for many hard to build packages. Part 2 > requires at least some changes in pip/PyPI. Precisely. But isn't part 2 at least sort-of solved by users manually pointing at "the right" index? The only files on PyPI are compatible with each other and externally hosted files (thanks for the pointer to the matplotlib binaries, BTW) won't get picked up automatically by pip so users have to set up their own index (possibly converting wininst->wheel) and so can manually manage the compatibility process if they are careful. If people start uploading incompatible binaries to PyPI, I expect a rash of bug reports followed very quickly by people settling down to a community-agreed standard (in fact, that's probably already happened). Incompatible builds will remain on external hosts like Cristoph's. It's not perfect, certainly, but it's no worse than currently. For any sort of better solution to part 2, you need *installed metadata* recording the ABI / shared library details for the installed files. So this is a Metadata 2.0 question, and not a compatibility tag / wheel issue (except that when Metadata 2.0 gets such information, Wheel 2.0 probably needs to be specified to validate against it or something). And on that note, I agree with Nick that we don't want to be going there at the moment, if ever. I just disagree with what I thought he was saying, that we should be so quick to direct people to conda (at some point we could debate why conda rather than ActiveState or Enthought, but tbh I really don't care...) I'd go with something along the lines of: """ Wheels don't attempt to solve the issue of one package depending on another one that has been built with specific options/compilers, or links to specific external libraries. The binaries on PyPI should always be compatible with each other (although nothing checks this, it's simply a matter of community standardisation), but if you use distributions hosted outside of PyPI or build your own, you need to manage such compatibility yourself. Most of the time, outside of specialised areas, it should not be an issue[1]. If you want guaranteed compatibility, you should use a distribution that validates and guarantees compatibility of all hosted files. This might be your platform package manager (apt or RPM) or a bundled Python distribution like Enthought, Conda or Activestate. """ [1] That statement is based on *my* experience. If problems are sufficiently widespread, we can tone it down, but let's not reach the point of FUD. Paul From p.f.moore at gmail.com Mon Dec 2 16:32:45 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 15:32:45 +0000 Subject: [Distutils] Package a project In-Reply-To: References: Message-ID: On 2 December 2013 15:28, Imran M Yousuf wrote: > Thanks for the suggestion Paul. Wheel structures exactly as I want it > to be, but I see it does not generate the entry point scripts; any > idea how to get them to work? The scripts should be generated when you install the wheel (with pip 1.5, which will be released soon). OTOH, I thought that wheel put generated scripts into the wheel file by default (except in version 0.20.0 - do you have the latest version?) The scripts should be in /.data/scripts - the wheel install process puts them in the right place (virtualenv/bin or whatever is appropriate). Paul From imran at smitsol.com Mon Dec 2 16:35:30 2013 From: imran at smitsol.com (Imran M Yousuf) Date: Mon, 2 Dec 2013 21:35:30 +0600 Subject: [Distutils] Package a project In-Reply-To: References: Message-ID: Damn me! Thanks Paul, yes the script does have it in the data folder. Thank you, Imran On Mon, Dec 2, 2013 at 9:32 PM, Paul Moore wrote: > On 2 December 2013 15:28, Imran M Yousuf wrote: >> Thanks for the suggestion Paul. Wheel structures exactly as I want it >> to be, but I see it does not generate the entry point scripts; any >> idea how to get them to work? > > The scripts should be generated when you install the wheel (with pip > 1.5, which will be released soon). OTOH, I thought that wheel put > generated scripts into the wheel file by default (except in version > 0.20.0 - do you have the latest version?) > > The scripts should be in /.data/scripts - the > wheel install process puts them in the right place (virtualenv/bin or > whatever is appropriate). > > Paul -- Imran M Yousuf Entrepreneur & CEO Smart IT Solution http://smitsol.com 25/5B, Block F, Haji Chinu Miah Road Bylane Joint Quarter, Mohammadpur Dhaka - 1207, Bangladesh Email: imran at smitsol.com Twitter: @imyousuf - http://twitter.com/imyousuf Skype: imyousuf Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557 +880-1746119494 From imran at smitsol.com Mon Dec 2 16:28:13 2013 From: imran at smitsol.com (Imran M Yousuf) Date: Mon, 2 Dec 2013 21:28:13 +0600 Subject: [Distutils] Package a project In-Reply-To: References: Message-ID: Thanks for the suggestion Paul. Wheel structures exactly as I want it to be, but I see it does not generate the entry point scripts; any idea how to get them to work? Thank you, Imran On Mon, Dec 2, 2013 at 7:59 PM, Paul Moore wrote: > On 2 December 2013 07:53, Imran M Yousuf wrote: >> Hi, >> >> I am new to setuptools. I am using it to build and package a project >> of mine. Currently if I execute `python setup.py bdist` it generates a >> tarball with all files located in paths >> './abs/path/to/project/bin/[entry points]' and >> './abs/path/to/project/lib/python-2.7/site-packages/[rest of the >> sources]'. This does not seem to be logical :(, I would rather want >> the binary distribution to be structure - >> './project-name/bin/' and './project-name/lib/'. >> >> Can some please advise me how to achieve it? I am using VirtualEnv for >> development of this project and its setup.py looks like - >> >> from setuptools import setup, find_packages >> >> setup(name='project-name', >> version='1.0', >> description='Description', >> author='Imran M Yousuf', >> author_email='imran at smitsol.com', >> url='http://www.smitsol.com', >> install_requires = ['setuptools', 'pycrypto==2.6'], >> packages=find_packages('src', ["tests"]), >> package_dir={'': 'src'}, >> test_suite="tests", >> entry_points={ >> 'console_scripts': ['manager=client.manager:main'] >> } >> ) > > Install the wheel project and use bdist_wheel instead of a simple > bdist. Also, use the sdist (source distribution) command to create a > source package (that needs a compiler to build). Binary packages are > only compatible with the platform/Python version they are built on, so > you may want to make multiple wheels, depending on what platforms you > are targeting. > > From what you provide, I'm not 100% sure if you have C code in your > project, actually. If you don't, then a sdist is sufficient - although > a wheel might be worth uploading as well (pure Python wheels are > cross-platform). > > The plain bdist command produces a "dumb" binary distribution, which > is obsolete, and frankly useless. > Paul -- Imran M Yousuf Entrepreneur & CEO Smart IT Solution http://smitsol.com 25/5B, Block F, Haji Chinu Miah Road Bylane Joint Quarter, Mohammadpur Dhaka - 1207, Bangladesh Email: imran at smitsol.com Twitter: @imyousuf - http://twitter.com/imyousuf Skype: imyousuf Blog: http://imyousuf-tech.blogs.smartitengineering.com/ Mobile: +880-1711402557 +880-1746119494 From vinay_sajip at yahoo.co.uk Mon Dec 2 18:23:02 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 2 Dec 2013 17:23:02 +0000 (GMT) Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: Message-ID: <1386004982.37213.YahooMailBasic@web171401.mail.ir2.yahoo.com> On Mon, 2/12/13, Tres Seaver wrote: > The issue is combinatorial explosion in the compatibility tag space. > There is basically zero chance that even Linux users (even RedHat > users across RHEL version) would benefit from pre-built binary > wheels (as opposed to packages from their distribution).? Wheels > on POSIX allow caching of the build process for deployment across > a known set of hosts: they won't insulate you from the need to build in > the first place. The combinations are number of Python X.Y versions x the no. of platform architectures/ABI variants, or do you mean something more than this? The wheel format is supposed to be a cross-platform binary package format; are you saying it is completely useless for POSIX except as a cache for identical hosts? What about for the cases like simple C extensions which have no external dependencies, but are only for speedups? What about POSIX environments where compilers aren't available (e.g. restricted/embedded environments, or due to security policies)? Regards, Vinay Sajip From tseaver at palladion.com Mon Dec 2 18:53:06 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 02 Dec 2013 12:53:06 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <1386004982.37213.YahooMailBasic@web171401.mail.ir2.yahoo.com> References: <1386004982.37213.YahooMailBasic@web171401.mail.ir2.yahoo.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/02/2013 12:23 PM, Vinay Sajip wrote: > On Mon, 2/12/13, Tres Seaver wrote: > >> The issue is combinatorial explosion in the compatibility tag >> space. There is basically zero chance that even Linux users (even >> RedHat users across RHEL version) would benefit from pre-built >> binary wheels (as opposed to packages from their distribution). >> Wheels on POSIX allow caching of the build process for deployment >> across a known set of hosts: they won't insulate you from the need >> to build in the first place. > > The combinations are number of Python X.Y versions x the no. of > platform architectures/ABI variants, or do you mean something more > than this? Trying to mark up wheels so that they can be safely shared with unknown POSIXy systems seems like a halting problem, to me: the chance I can build a wheel on my machine that you can use on yours (the only reason to distribute a wheel, rather than the sdist, in the first place) drops off sharply as wheel's "binariness" comes into play. I'm arguing that wheel is not an interesting *distribution* format for POSIX systems (at least, for non-Mac ones). It could still play out in *deployment* scenarios (as you note below). Note that wheel's main deployment advantage over a binary egg (installable by pip) is exactly reversed if you use 'easy_install' or 'zc.buildout'. Otherwise, in a controlled deployment, they are pretty much equivalent. > The wheel format is supposed to be a cross-platform binary package > format; are you saying it is completely useless for POSIX except as a > cache for identical hosts? What about for the cases like simple C > extensions which have no external dependencies, but are only for > speedups? I have a lot of packages on PyPI which have such optimization-only speeedups. The time difference to build such extensions is trivial (e.g., for zope.interface, ~1 second on my old slow laptop, versus 0.4 seconds without the extension). Even for lxml (Daniel's original motivating case), the difference is ~45 seconds to build from source vs. 1 second to install a wheel (or and egg). The instant I have to think about whether the binary form might be subtly incompatbile, that 1 second *loses* to the 45 seconds I spend over here arguing with you guys while it builds again from source. :) > What about POSIX environments where compilers aren't available (e.g. > restricted/embedded environments, or due to security policies)? Such environments are almost certainly driven by development teams who can build wheels specifically for deployment to them (assuming the policies allow anything other than distro-package-managed software). This is still really a "cache the build" optimization to known platforms (w/ all binary dependencies the same), rather than distribution. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKcyPsACgkQ+gerLs4ltQ4oBwCgvhoq8ovEn/Bl/0FpBEfI48JY znEAoJElD+R9SPnJXduwjCy7oxWRmcWH =a0TT -----END PGP SIGNATURE----- From qwcode at gmail.com Mon Dec 2 18:56:28 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 2 Dec 2013 09:56:28 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > hash based dependencies > In the conda build guide, the yaml spec files reference dependencies by name/version (and the type of conda environment you're in will determine the rest) http://docs.continuum.io/conda/build.html#specifying-versions-in-requirements Where does the hash come in? what do you mean? publication of curated stacks when the conda folks already have one, > so, I see the index: http://repo.continuum.io/pkgs/index.html Is they a way to contribute to this index yet? or is that what would need to be worked out. otherwise, I guess the option is you have to build out "recipes" for anything else you need from pypi, right? or is it easier than that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Dec 2 19:03:01 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 2 Dec 2013 10:03:01 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > In the conda build guide, the yaml spec files reference dependencies by > name/version (and the type of conda environment you're in will determine > the rest) > > http://docs.continuum.io/conda/build.html#specifying-versions-in-requirements > Where does the hash come in? what do you mean? > e.g. here's the requirement section from the spec file for their recipe for fabric. https://github.com/ContinuumIO/conda-recipes/blob/master/fabric/meta.yaml#L28 requirements: build: - python - distribute - paramiko run: - python - distribute - paramiko -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Mon Dec 2 20:17:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 2 Dec 2013 14:17:42 -0500 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> Message-ID: On Mon, Dec 2, 2013 at 9:14 AM, Michael Jansen wrote: > Hi > > > > I am currently working on the cobbler (http://cobblerd.org) setup.py and > trying to improve it. It currently is only capable of installing to /usr for > several reasons. Since i would like to have virtualenv support when running > it i am trying to change that. While doing that i managed to meet one of the > standard problems with distutils/setuptools - a search returns results of > 2004 or older - which still seems to be unsolved. > > > > I would be willing to change that but my recent research leaves me a bit > confused with all that setuptools is not dead, distlibs, tuf and wheels > stuff. > > > > So here is the question. If i am willing to do the work is it possible to > add something like sbin_scripts (or admin_scripts?) to distutils? > > > > I locally solved the problem like that http://pastebin.kde.org/pqrwrud1p (or > attached). > > > > The implementation expands INSTALL_SCHEMA and reuses install_scripts to > install those sbin_scripts too. That could be debatable. Perhaps a dedicated > build|install_xxx_scripts command. > > > > I want to add that fhs also has a libexec folder (usually > /usr/lib//libexec these days) destined for scripts that are > supposed to be only called by other scripts/programs not manually. > > > > And cobbler for installs python scripts for modwsgi into /srv/www/cobbler/ . > So i guess just adding sbin_scripts is not really the solution. Perhaps > something more flexible is needed. > > > > Or should i wait for nextgen distlib based swiss knife python build tool and > for now keep my local additions (there are more). > > > > -- > > Michael Jansen > > http://michael-jansen.biz It would be fairly simple to propose adding the GNU coding standard directory variables: http://www.gnu.org/prep/standards/html_node/Directory-Variables.html . They could become valid names under the wheel .data/ directory. You would also want to figure out how to instruct the installer to generate scripts wrappers for these alternative destinations so e.g. the execute bit is set and the #! line points to the correct interpreter. It would be a good idea to consider the needs of Windows and virtualenv users who may need to be able to opt out of the "install things anywhere on my system" and "full FHS compliance" features. From erik.m.bray at gmail.com Mon Dec 2 20:48:58 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 2 Dec 2013 14:48:58 -0500 Subject: [Distutils] How setuptools reads requirements In-Reply-To: References: Message-ID: On Tue, Nov 19, 2013 at 1:56 PM, Kura wrote: > Hey guys > > I'm trying to dig in to how setuptools/distutils reads the > [install_]requires keyword and how requirements.txt is handled. > > I've tried running setup.py --requires but get an empty response back, this > also fails when using >=x.x,<=x.y and causes a Python stack trace. > > Was wondering if someone on here could shed some light on the topic for me. `setup.py --requires` is mostly useless. It only reports back entries from the `requires=` argument to setup() supported by distutils. This only tracks the names of other modules that a module needs to import, and is being phased out as not particularly useful. It's not in any way tied to `install_requires` which is a feature added by setuptools. As for "requirements.txt", you might mean "requires.txt" which is a file added to the .egg_info directory when you run the `setup.py egg_info` command on a setuptools-enabled package. Pip has its own code for reading information from files out of .egg_info. I believe there are other libraries like pkginfo [1] that do this. > It's for a Python dependency management/monitoring system I am writing so > hopefully would benefit the overall community. Sounds intriguing! [1] https://pypi.python.org/pypi/pkginfo From ncoghlan at gmail.com Mon Dec 2 22:44:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 07:44:57 +1000 Subject: [Distutils] Package a project In-Reply-To: References: Message-ID: On 3 Dec 2013 02:01, "Imran M Yousuf" wrote: > > Thanks for the suggestion Paul. Wheel structures exactly as I want it > to be, but I see it does not generate the entry point scripts; any > idea how to get them to work? Those are platform dependent, so the installer generates them at install time based on the metadata in the wheel. Cheers, Nick. > > Thank you, > > Imran > > On Mon, Dec 2, 2013 at 7:59 PM, Paul Moore wrote: > > On 2 December 2013 07:53, Imran M Yousuf wrote: > >> Hi, > >> > >> I am new to setuptools. I am using it to build and package a project > >> of mine. Currently if I execute `python setup.py bdist` it generates a > >> tarball with all files located in paths > >> './abs/path/to/project/bin/[entry points]' and > >> './abs/path/to/project/lib/python-2.7/site-packages/[rest of the > >> sources]'. This does not seem to be logical :(, I would rather want > >> the binary distribution to be structure - > >> './project-name/bin/' and './project-name/lib/'. > >> > >> Can some please advise me how to achieve it? I am using VirtualEnv for > >> development of this project and its setup.py looks like - > >> > >> from setuptools import setup, find_packages > >> > >> setup(name='project-name', > >> version='1.0', > >> description='Description', > >> author='Imran M Yousuf', > >> author_email='imran at smitsol.com', > >> url='http://www.smitsol.com', > >> install_requires = ['setuptools', 'pycrypto==2.6'], > >> packages=find_packages('src', ["tests"]), > >> package_dir={'': 'src'}, > >> test_suite="tests", > >> entry_points={ > >> 'console_scripts': ['manager=client.manager:main'] > >> } > >> ) > > > > Install the wheel project and use bdist_wheel instead of a simple > > bdist. Also, use the sdist (source distribution) command to create a > > source package (that needs a compiler to build). Binary packages are > > only compatible with the platform/Python version they are built on, so > > you may want to make multiple wheels, depending on what platforms you > > are targeting. > > > > From what you provide, I'm not 100% sure if you have C code in your > > project, actually. If you don't, then a sdist is sufficient - although > > a wheel might be worth uploading as well (pure Python wheels are > > cross-platform). > > > > The plain bdist command produces a "dumb" binary distribution, which > > is obsolete, and frankly useless. > > Paul > > > > -- > Imran M Yousuf > Entrepreneur & CEO > Smart IT Solution > http://smitsol.com > 25/5B, Block F, Haji Chinu Miah Road Bylane > Joint Quarter, Mohammadpur > Dhaka - 1207, Bangladesh > Email: imran at smitsol.com > Twitter: @imyousuf - http://twitter.com/imyousuf > Skype: imyousuf > Blog: http://imyousuf-tech.blogs.smartitengineering.com/ > Mobile: +880-1711402557 > +880-1746119494 > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Mon Dec 2 23:17:24 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 2 Dec 2013 14:17:24 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > publication of curated stacks when the conda folks already have one, >> > so, I see the index: http://repo.continuum.io/pkgs/index.html > Is they a way to contribute to this index yet? or is that what would need > to be worked out. > probably a dumb question, but would it be possible to convert all the anaconda packages to wheels? even the non-python ones like: qt-4.7.4-0.tar.bz2 certainly not the intent of wheels, but just wondering if it could be made to work? but I'm guessing there's pieces in the core anaconda distribution itself, that makes it all work? the point here being to provide a way to use the effort of conda in any kind of "normal" python environment, as long you consistently point at an index that just contains the "conda" wheels. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 2 23:26:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 08:26:02 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 Dec 2013 00:19, "Paul Moore" wrote: > > On 2 December 2013 13:38, Tres Seaver wrote: > > On 12/01/2013 06:38 PM, Paul Moore wrote: > >> I understand that things are different in the Unix world, but to be > >> blunt why should Windows users care? > > > > You're kidding, right? 90% or more of the reason for wheels in the first > > place is because Windows users can't build their own software from > > source. The amount of effort put in by non-Windows package owners to > > support them dwarfs whatever is bothering you here. > > My point is that most of the complex binary compatibility problems > seem to be Unix-related, and as you imply, Unix users don't seem to > have much interest in using wheels except for local caching. So why > build that complexity into the spec if the main users (Windows, and > Unix users who won't ever publish wheels outside their own systems) > don't have a need for it? Let's just stick with something simple that > has limitations but works (practicality beats purity). My original > bdist_simple proposal was a pure-Windows replacement for wininst. > Daniel developed that into wheels which cater for non-Windows systems > (I believe, precisely because he had an interest in the local cache > use case). We're now seeing the complexities of the Unix world affect > the design of wheels, and it's turning out to be a hard problem. All > I'm trying to say is let's not give up on binary wheels for Windows, > just because we have unsolved issues on Unix. Huh? This is *exactly* what I am saying we should do - wheels *already* work so long as they're self-contained. They *don't* work (automatically) when they have an external dependency: users have to obtain the external dependency by other means, and ensure that everything is properly configured to find it, and that everything is compatible with the retrieved version. You're right that Christoph is doing two different things, though, so our advice to him (or anyone that wanted to provide the cross-platform equivalent of his current Windows-only stack) would be split: - for all self-contained installers, also publish a wheel file on a custom index server (although having a "builder" role on PyPI where project owners can grant someone permission to upload binaries after the sdist is published could be interesting) - for those installers which actually form an integrated stack with shared external binary dependencies, use the mechanisms provided by conda rather than getting users to manage the external dependencies by hand (as licensing permits, anyway) >Whether solving the Unix > issues is worth it is the Unix users' call - I'll help solve the > issues, if they choose to, but I won't support abandoning the existing > Windows solution just because it can't be extended to cater for Unix > as well. You appear to still be misunderstanding my proposal, as we're actually in violent agreement. All that extra complexity you're worrying about is precisely what I'm saying we should *leave out* of the wheel spec. In most cases of accelerator and wrapper modules, the static linking and/or bundling solutions will work fine, and that's the domain I believe we should *deliberately* restrict wheels to, so we don't get distracted trying to solve an incredibly hard external dependency management problem that we don't actually need to solve at the wheel level, since anyone that actually needs it solved can just bootstrap conda instead. > I'm immensely grateful for the amount of work projects which are > developed on Unix (and 3rd parties like Cristoph) put into supporting > Windows. Far from dismissing that, I want to avoid making things any > harder than they already are for such people - current wheels are no > more complex to distribute than wininst installers, and I want to keep > the impact on non-Windows projects at that level. If I come across as > ungrateful, I apologise. The only problem I want to explicitly declare out of scope for wheel files is the one the wininst installers can't handle cleanly either: the subset of Christoph's installers which need a shared external binary dependency, and any other components in a similar situation. Using wheels or native Windows installers can get you in trouble in that case, since you may accidentally set up conflicts in your environment. The solution is curation of a software stack built around that external dependency (or dependencies), backed up by a packaging system that prevents conflicts within a given local installation. The mainstream Linux distros approach this problem by mapping everything to platform-specific packages and trying to get parallel installation working cleanly (a part of the problem I plan to work on improving post Python 3.4), but that approach doesn't scale well and is one of the factors responsible for the notorious time lags between software being released on PyPI and it being available in the Linux system package managers (streamlining that conversion is one of my main goals for metadata 2.0). The conda folks approach it differently by using hashes to lock in *exact* versions of dependencies. From a compatibility management perspective, it's functionally equivalent to bundling a complete stack into a single monolithic installer, but has the benefit of still making it easy to only install the pieces that you actually need locally. Regards, Nick. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Dec 3 00:03:12 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 2 Dec 2013 23:03:12 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 2 December 2013 22:26, Nick Coghlan wrote: >>Whether solving the Unix >> issues is worth it is the Unix users' call - I'll help solve the >> issues, if they choose to, but I won't support abandoning the existing >> Windows solution just because it can't be extended to cater for Unix >> as well. > > You appear to still be misunderstanding my proposal, as we're actually in > violent agreement. All that extra complexity you're worrying about is > precisely what I'm saying we should *leave out* of the wheel spec. In most > cases of accelerator and wrapper modules, the static linking and/or bundling > solutions will work fine, and that's the domain I believe we should > *deliberately* restrict wheels to, so we don't get distracted trying to > solve an incredibly hard external dependency management problem that we > don't actually need to solve at the wheel level, since anyone that actually > needs it solved can just bootstrap conda instead. OK. I think I've finally seen what you're suggesting, and yes, it's essentially the same as I'd like to see (at least for now). I'd hoped that wheels could be more useful for Unix users than seems likely now - mainly because I really do think that a lot of the benefits of binary distributions are *not* restricted to Windows, and if Unix users could use them, it'd lessen the tendency to think that supporting anything other than source installs was purely "to cater for Windows users not having a compiler" :-) But if that's not a practical possibility (and I defer to the Unix users' opinions on that matter) then so be it. On the other hand, I still don't see where the emphasis on conda in your original message came from. There are lots of "full stack" solutions available - I'd have thought system packages like RPM and apt are the obvious first suggestion for users that need a curated stack. If they are not appropriate, then there are Enthought, ActiveState and Anaconda/conda that I know of. Why single out conda to be blessed? Also, I'd like the proposal to explicitly point out that 99% of the time, Windows is the simple case (because static linking and bundling DLLs is common). Getting Windows users to switch to wheels will be enough change to ask, without confusing the message. A key point here is that packages like lxml, matplotlib, or Pillow would have "arbitrary binary dependency issues" on Unix, but (because of static linking/bundling) be entirely appropriate for wheels on Windows. Let's make sure the developers don't miss this point! Paul From ncoghlan at gmail.com Mon Dec 2 23:57:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 08:57:48 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 Dec 2013 08:17, "Marcus Smith" wrote: > > >>> >>> publication of curated stacks when the conda folks already have one, >> >> so, I see the index: http://repo.continuum.io/pkgs/index.html >> Is they a way to contribute to this index yet? or is that what would need to be worked out. > > > probably a dumb question, but would it be possible to convert all the anaconda packages to wheels? > even the non-python ones like: qt-4.7.4-0.tar.bz2 > certainly not the intent of wheels, but just wondering if it could be made to work? > but I'm guessing there's pieces in the core anaconda distribution itself, that makes it all work? > the point here being to provide a way to use the effort of conda in any kind of "normal" python environment, as long you consistently point at an index that just contains the "conda" wheels. I'm not sure about the conda -> wheel direction, but "pip install conda && conda init" mostly works already if you're in a virtualenv that owns its copy of Python (this is also the answer to "why not ActiveState or Enthought" - the Continuum Analytics software distribution stuff is truly open source, and able to be used completely independently of their services). Their docs aren't that great in terms of explaining the *why* of conda - I'm definitely influenced by spending time talking about how it works with Travis and some of the other Continuum Analytics folks at PyCon US and the Austin Python user group. However, their approach to distribution of fully curated stacks seems basically sound, the scientific and data analysis users I know that have tried it have loved it, the devs have expressed a willingness to work on improving their interoperability with the standard tools (and followed through on that at least once by creating the "conda init" command) , and they're actively interested in participating in the broader community (hence the presentation at the packaging mini-summit at PyCon US, as well as assorted presentations at SciPy and PyData conferences). People are already confused about the differences between pip and conda and when they should use each, and unless we start working with the conda devs to cleanly define the different use cases, that's going to remain the case. POSIX users need ready access to a prebuilt scientific stack just as much (or more) than Mac OS X and Windows users (there's a reason "ScientificLinux" is a distribution in its own right) and that space is moving fast enough that the Linux distros (even SL) end up being too slow to update. conda solves that problem, and it solves it in a way that works on Windows as well. On the wheel side of things we haven't even solved the POSIX platform tagging problem yet, and I don't believe we should make users wait until we have figured that out when there's an existing solution to that particular problem that already works. Cheers, Nick. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Dec 3 00:54:43 2013 From: qwcode at gmail.com (Marcus Smith) Date: Mon, 2 Dec 2013 15:54:43 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > I'm not sure about the conda -> wheel direction, but "pip install conda && > conda init" mostly works already if you're in a virtualenv that owns its > copy of Python > ok, I just tried conda in a throw-away altinstall of py2.7. I was thinking I would have to "conda create" new isolated environments from there. but there literally is a "conda init" (*not* documented on the website) like you mentioned that get's conda going in the current environment. pip and conda were both working, except that pip didn't know about everything conda had installed like sqllite, which is expected. and I found all the conda metadata which was helpful to look at. I still don't know what you mean by "hash based dependencies". I'm not seeing any requirements being locked by hashes in the metadata? what do you mean? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 3 00:48:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 09:48:36 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 Dec 2013 09:03, "Paul Moore" wrote: > > On 2 December 2013 22:26, Nick Coghlan wrote: > >>Whether solving the Unix > >> issues is worth it is the Unix users' call - I'll help solve the > >> issues, if they choose to, but I won't support abandoning the existing > >> Windows solution just because it can't be extended to cater for Unix > >> as well. > > > > You appear to still be misunderstanding my proposal, as we're actually in > > violent agreement. All that extra complexity you're worrying about is > > precisely what I'm saying we should *leave out* of the wheel spec. In most > > cases of accelerator and wrapper modules, the static linking and/or bundling > > solutions will work fine, and that's the domain I believe we should > > *deliberately* restrict wheels to, so we don't get distracted trying to > > solve an incredibly hard external dependency management problem that we > > don't actually need to solve at the wheel level, since anyone that actually > > needs it solved can just bootstrap conda instead. > > OK. I think I've finally seen what you're suggesting, and yes, it's > essentially the same as I'd like to see (at least for now). I'd hoped > that wheels could be more useful for Unix users than seems likely now > - mainly because I really do think that a lot of the benefits of > binary distributions are *not* restricted to Windows, and if Unix > users could use them, it'd lessen the tendency to think that > supporting anything other than source installs was purely "to cater > for Windows users not having a compiler" :-) But if that's not a > practical possibility (and I defer to the Unix users' opinions on that > matter) then so be it. > > On the other hand, I still don't see where the emphasis on conda in > your original message came from. There are lots of "full stack" > solutions available - I'd have thought system packages like RPM and > apt are the obvious first suggestion for users that need a curated > stack. If they are not appropriate, then there are Enthought, > ActiveState and Anaconda/conda that I know of. Why single out conda to > be blessed? > > Also, I'd like the proposal to explicitly point out that 99% of the > time, Windows is the simple case (because static linking and bundling > DLLs is common). Getting Windows users to switch to wheels will be > enough change to ask, without confusing the message. A key point here > is that packages like lxml, matplotlib, or Pillow would have > "arbitrary binary dependency issues" on Unix, but (because of static > linking/bundling) be entirely appropriate for wheels on Windows. Let's > make sure the developers don't miss this point! Once we solve the platform tagging problem, wheels will also work on any POSIX system for the simple cases of accelerator and wrapper modules. Long term the only persistent problem is with software stacks that need consistent build settings and offer lots of build options. That applies to Windows as well - the SSE build variants of NumPy were one of the original cases brought up as not being covered by the wheel compatibility tag format. Near term, platform independent stacks *also* serve as a workaround for the POSIX platform tagging issues and the fact there isn't yet a "default" build configuration for the scientific stack. As for "Why conda?": - open source - cross platform - can be installed with pip - gets new releases of Python components faster than Linux distributions - uses Continuum Analytics services by default, but can be configured to use custom servers - created by the creator of NumPy For ActiveState and Enthought, as far as I am aware, their package managers are closed source and tied fairly closely to their business model, while the Linux distros are not only platform specific, but have spotty coverage of PyPI packages, and even those which are covered, often aren't reliably kept up to date (although I hope metadata 2.0 will help improve that situation by streamlining the conversion to policy compliant system packages). Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Dec 3 01:27:34 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 2 Dec 2013 16:27:34 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Mon, Dec 2, 2013 at 5:22 AM, Nick Coghlan wrote: > And the conda folks are working on playing nice with virtualenv - I don't > we'll see a similar offer from Microsoft for MSI any time soon :) > nice to know... > > > a single organisation. Pip (when used normally) communicates with PyPI > > > and no single organisation controls the content of PyPI. > can't you point pip to a "wheelhouse'? How is that different? >>For built distributions they could do > > > the same - except that pip/PyPI don't provide a mechanism for them to > > > do so. > I'm still confused as to what conda provides here -- as near as I can tell, conda has a nice hash-based way to ensure binary compatibility -- which is a good thing. But the "curated set of packages" is an independent issue. What's stopping anyone from creating a nice curated set of packages with binary wheels (like the Gohlke repo....) And wouldn't it be better to make wheel a bit more robust in this regard than add yet another recommended tool to the mix? > Exactly, this is the difference between pip and conda - conda is a > solution for installing from curated *collections* of packages. It's > somewhat related to the tagging system people are speculating about for > PyPI, but instead of being purely hypothetical, it already exists. > Does it? I only know of one repository of conda packages -- and it provides poor support for some things (like wxPython -- does it support any desktop GUI on OS-X?) So why do we think that conda is a better option for these unknown curatied repos? Also, I'm not sure I WANT anymore curated repos -- I'd rather a standard set by python.org that individual package maintainers can choose to support. PyPI wheels would then be about publishing "default" versions of > components, with the broadest compatibility, while conda would be a > solution for getting access to alternate builds that may be faster, but > require external shared dependencies. > I'm still confused as to why packages need to share external dependencies (though I can see why it's nice...) . But what's the new policy here? Anaconda and Canopy exist already? Do we need to endorse them? Why? If you want "PyPI wheels would then be about publishing "default" versions of components, with the broadest compatibility," -- then we still need to improve things a bit, but we can't say "we're done" What Christoph is doing is producing a cross-platform curated binary > software stack, including external dependencies. That's precisely the > problem I'm suggesting we *not* try to solve in the core tools any time > soon, but instead support bootstrapping conda to solve the problem at a > different layer. > So we are advocating that others, like Christoph, create curated stack with conda? Asside from whether conda really provides much more than wheel to support doing this, I think it's a BAD idea to encourage it: I'd much rather encourage package maintainers to build "standard" packages, so we can get some extra interoperabilty. Example: you can't use wxPython with Anocoda (on the Mac, anyway). At least not without figuring out how to build it yourself, an I'm not sure it will even work then. (and it is a fricking nightmare to build). But it's getting harder to find "standard" packages for the mac for the SciPy stack, so people are really stuck. So the pip compatible builds for those tools would likely miss out on some > of the external acceleration features, > that's fine -- but we still need those pip compatible builds .... and the nice thing about pip-compatible builds (really python.orgcompatible builds...) is that they play well with the other binary installers -- > By ceding the "distribution of cross-platform curated software stacks with > external binary dependencies" problem to conda, users would get a solution > to that problem that they can use *now*, > Well, to be fair, I've been starting a project to provide binaries for various packages for OS_X amd did intend to give conda a good look-see, but I really has hoped that wheels where "the way" now...oh well. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at michael-jansen.biz Tue Dec 3 09:41:27 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Tue, 03 Dec 2013 09:41:27 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> Message-ID: <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> > It would be fairly simple to propose adding the GNU coding standard > directory variables: > http://www.gnu.org/prep/standards/html_node/Directory-Variables.html . > They could become valid names under the wheel .data/ directory. It probably would. But i am a fan of only adding whats now needed. So i think just adding support for /sbin and /libexec would be enough for the problem i have at hand NOW. I need support for /etc to and the special handling for /usr/etc -> /etc too for some other stuff but i would like to do it one thing after the other. Lets first find out if its desired and possible to add something to distuils. It still is the base of all other build systems? Right? After that perhaps adding those to setuptools scripts entry point should be the next step. THEN i start to think about what i need for other stuff :) . > You would also want to figure out how to instruct the installer to > generate scripts wrappers for these alternative destinations so e.g. > the execute bit is set and the #! line points to the correct > interpreter. Thats what my patch does. For /sbin for now but support of the libexec directory is easy to add. But quite a bit of copy and paste programming. So another idea i am thinking about is doing it like this: scripts = [ "bin/myscript3", # <= backwardscompatible ( 'bin/', [ 'bin/myscript1', 'bin/myscript2' ] ), ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( 'lib/myproject/libexec', [ 'libexec/script' ] ) ] So if its a string do it like always and if its a list/tuple do it like data_files does. In an ideal world one could even expand those gnu standard vars in there. Like that: scripts = [ "bin/myscript3", # <= backwardscompatibility ( '$bindir/', [ 'bin/myscript1', 'bin/myscript2' ] ), ( '$sbindir/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), ( '$libexecdir', [ 'libexec/script' ] ) ] > It would be a good idea to consider the needs of Windows and > virtualenv users who may need to be able to opt out of the "install > things anywhere on my system" and "full FHS compliance" features. At least the need for Windows is taken care of by INSTALL_SCHEME in distutils.command.install . The sbin directory is added like that: if sys.version < "2.2": dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' else: dist_install.WINDOWS_SCHEME['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['unix_prefix']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_home']['sbin_scripts'] = '$base/sbin' dist_install.INSTALL_SCHEMES['unix_user']['sbin_scripts'] = '$userbase/sbin' dist_install.INSTALL_SCHEMES['nt_user']['sbin_scripts'] = '$userbase/Scripts' dist_install.INSTALL_SCHEMES['os2']['sbin_scripts'] = '$base/Scripts' dist_install.INSTALL_SCHEMES['os2_home']['sbin_scripts'] = '$userbase/sbin' As you can see for windows the sbin scripts get installed into Scripts too. Same for os2 unless the --home option is active. And virtualenv does not need any kind of special handling. It has to install the same just into a different prefix. Same should go for pyvenv ? . All those new directories just have to be added there with correct values for all schemes. Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 3 09:48:41 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 18:48:41 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: Thanks for the robust feedback folks - it's really helping me to clarify what I think, and why I consider this an important topic :) On 3 Dec 2013 10:36, "Chris Barker" wrote: > > On Mon, Dec 2, 2013 at 5:22 AM, Nick Coghlan wrote: >> >> And the conda folks are working on playing nice with virtualenv - I don't we'll see a similar offer from Microsoft for MSI any time soon :) > > nice to know... >> >> > > a single organisation. Pip (when used normally) communicates with PyPI >> > > and no single organisation controls the content of PyPI. > > can't you point pip to a "wheelhouse'? How is that different? Right, you can do integrated environments with wheels, that's one of the use cases they excel at. > >> >>For built distributions they could do >> > > the same - except that pip/PyPI don't provide a mechanism for them to >> > > do so. > > I'm still confused as to what conda provides here -- as near as I can tell, conda has a nice hash-based way to ensure binary compatibility -- which is a good thing. But the "curated set of packages" is an independent issue. What's stopping anyone from creating a nice curated set of packages with binary wheels (like the Gohlke repo....) Hmm, has anyone tried running devpi on a PaaS? :) > And wouldn't it be better to make wheel a bit more robust in this regard than add yet another recommended tool to the mix? Software that works today is generally more useful to end users than software that might possibly handle their use case at some currently unspecified point in the future :) >> Exactly, this is the difference between pip and conda - conda is a solution for installing from curated *collections* of packages. It's somewhat related to the tagging system people are speculating about for PyPI, but instead of being purely hypothetical, it already exists. > > Does it? I only know of one repository of conda packages -- and it provides poor support for some things (like wxPython -- does it support any desktop GUI on OS-X?) > > So why do we think that conda is a better option for these unknown curatied repos? Because it already works for the scientific stack, and if we don't provide any explicit messaging around where conda fits into the distribution picture, users are going to remain confused about it for a long time. > Also, I'm not sure I WANT anymore curated repos -- I'd rather a standard set by python.org that individual package maintainers can choose to support. > >> PyPI wheels would then be about publishing "default" versions of components, with the broadest compatibility, while conda would be a solution for getting access to alternate builds that may be faster, but require external shared dependencies. > > I'm still confused as to why packages need to share external dependencies (though I can see why it's nice...) . Because they reference shared external data, communicate through shared memory, or otherwise need compatible memory layouts. It's exactly the same reason all C extensions need to be using the same C runtime as CPython on Windows: because things like file descriptors break if they don't. > But what's the new policy here? Anaconda and Canopy exist already? Do we need to endorse them? Why? If you want "PyPI wheels would then be about publishing "default" versions of components, with the broadest compatibility," -- then we still need to improve things a bit, but we can't say "we're done" Conda solves a specific problem for the scientific community, but in their enthusiasm, the developers are pitching it as a general purpose packaging solution. It isn't, but in the absence of a clear explanation of its limitations from us, both its developers and other Python users are likely to remain confused about the matter. > >> What Christoph is doing is producing a cross-platform curated binary software stack, including external dependencies. That's precisely the problem I'm suggesting we *not* try to solve in the core tools any time soon, but instead support bootstrapping conda to solve the problem at a different layer. > > So we are advocating that others, like Christoph, create curated stack with conda? Asside from whether conda really provides much more than wheel to support doing this, I think it's a BAD idea to encourage it: I'd much rather encourage package maintainers to build "standard" packages, so we can get some extra interoperabilty. > > Example: you can't use wxPython with Anocoda (on the Mac, anyway). At least not without figuring out how to build it yourself, an I'm not sure it will even work then. (and it is a fricking nightmare to build). But it's getting harder to find "standard" packages for the mac for the SciPy stack, so people are really stuck. > >> So the pip compatible builds for those tools would likely miss out on some of the external acceleration features, > > that's fine -- but we still need those pip compatible builds .... > > and the nice thing about pip-compatible builds (really python.orgcompatible builds...) is that they play well with the other binary installers -- >> >> By ceding the "distribution of cross-platform curated software stacks with external binary dependencies" problem to conda, users would get a solution to that problem that they can use *now*, > > Well, to be fair, I've been starting a project to provide binaries for various packages for OS_X amd did intend to give conda a good look-see, but I really has hoped that wheels where "the way" now...oh well. Wheels *are* the way if one or both of the following conditions hold: - you don't need to deal with build variants - you're building for a specific target environment That covers an awful lot of ground, but there's one thing it definitely doesn't cover: distributing multiple versions of NumPy built with different options and cohesive ecosystems on top of that. Now, there are various ideas for potentially making wheels handle that, from the scientific community all agreeing on a common set of build settings and publishing consistent wheels, to a variant tagging system, to having pip remember the original source of a software distribution, but they're all vapourware at this point, with no concrete plans to change that, and plenty of higher priority problems to deal with. By contrast, conda already exists, and already works, as it was designed *specifically* to handle the scientific Python stack. Unfortunately, the folks making conda don't quite grasp the full breadth of the use cases that pip handles, so their docs do a lousy job of explaining conda's limitations. It meets their needs perfectly though, along with the needs of many other people, so they're correspondingly enthusiastic in wanting to share its benefits with others. This means that one key reason I want to recommend it for the cases where it is a good fit (i.e. the scientific Python stack) is so we can explicitly advise *against* using it in other cases where it will just add complexity without adding value. Saying nothing is not an option, since people are already confused. Saying to never use it isn't an option either, since bootstrapping conda first *is* a substantially simpler cross-platform way to get up to date scientific Python software on to your system. The alternatives are platform specific and (at least in the Linux distro case) slower to get updates. Cheers, Nick. > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Dec 3 09:56:26 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 08:56:26 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: On 3 December 2013 08:41, Michael Jansen wrote: > So another idea i am thinking about is doing it like this: > > > > scripts = [ > > "bin/myscript3", # <= backwardscompatible > > ( 'bin/', [ 'bin/myscript1', 'bin/myscript2' ] ), > > ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), > > ( 'lib/myproject/libexec', [ 'libexec/script' ] ) > > ] The "scripts" argument is generally considered obsolete - tools like pip will not generate executable wrappers for scripts specified like this, which means that they will not work properly on Windows. You should really be using (in this case, extending) the setuptools "console_scripts" entry point. Paul From p.f.moore at gmail.com Tue Dec 3 10:11:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 09:11:59 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 08:48, Nick Coghlan wrote: >> And wouldn't it be better to make wheel a bit more robust in this regard >> than add yet another recommended tool to the mix? > > Software that works today is generally more useful to end users than > software that might possibly handle their use case at some currently > unspecified point in the future :) See my experience with conda under Windows. While I'm not saying that conda "doesn't work", being directed to software that turns out to have its own set of bugs, different to the ones you're used to, is a pretty frustrating experience. (BTW, I raised a bug report. Let's see what the response is like...) Paul From p.f.moore at gmail.com Tue Dec 3 10:22:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 09:22:59 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 08:48, Nick Coghlan wrote: > This means that one key reason I want to recommend it for the cases where it > is a good fit (i.e. the scientific Python stack) is so we can explicitly > advise *against* using it in other cases where it will just add complexity > without adding value. > > Saying nothing is not an option, since people are already confused. Saying > to never use it isn't an option either, since bootstrapping conda first *is* > a substantially simpler cross-platform way to get up to date scientific > Python software on to your system. The alternatives are platform specific > and (at least in the Linux distro case) slower to get updates. But you're not saying "use conda for the scientific Python stack". You're saying to use it "when you have binary external dependencies" which is a phrase that I (and I suspect many Windows users) don't really understand and will take to mean "C extensions, or at least ones that interface to another library, sich as pyyaml, lxml, ...) Also, this presumes an either/or situation. What about someone who just wants to use matplotlib to display a graph of some business data? Is matplotlib part of "the scientific stack"? Should I use conda *just* to get matplotlib in an otherwise wheel-based application? Or how about a scientist that wants wxPython (to use Chris' example)? Apparently the conda repo doesn't include wxPython, so do they need to learn how to install pip into a conda environment? (Note that there's no wxPython wheel, so this isn't a good example yet, but I'd hope it will be in due course...) Reducing confusion is good, I'm all for that. But we need to have a clear picture of what we're saying before we can state it clearly... Paul From p.f.moore at gmail.com Tue Dec 3 10:34:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 09:34:44 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 09:11, Paul Moore wrote: > On 3 December 2013 08:48, Nick Coghlan wrote: >>> And wouldn't it be better to make wheel a bit more robust in this regard >>> than add yet another recommended tool to the mix? >> >> Software that works today is generally more useful to end users than >> software that might possibly handle their use case at some currently >> unspecified point in the future :) > > See my experience with conda under Windows. While I'm not saying that > conda "doesn't work", being directed to software that turns out to > have its own set of bugs, different to the ones you're used to, is a > pretty frustrating experience. (BTW, I raised a bug report. Let's see > what the response is like...) Looks like the conda stack is built around msvcr90, whereas python.org Python 3.3 is built around msvcr100. So conda is not interoperable *at all* with standard python.org Python 3.3 on Windows :-( Paul From p.f.moore at gmail.com Tue Dec 3 10:37:03 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 09:37:03 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <40770026.o8Us6A34yE@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> <40770026.o8Us6A34yE@gambit.local.michael-jansen.biz> Message-ID: On 3 December 2013 09:38, Michael Jansen wrote: > So who will decide if any change to distutils goes through? I have no idea > about the python patch process. Any links? Patches can be posted on http://bugs.python.org. But generally, changes to distutils are hard to get through so be prepared for some pushback. Paul From info at michael-jansen.biz Tue Dec 3 10:38:51 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Tue, 03 Dec 2013 10:38:51 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: <40770026.o8Us6A34yE@gambit.local.michael-jansen.biz> On Tuesday, December 03, 2013 08:56:26 AM Paul Moore wrote: > On 3 December 2013 08:41, Michael Jansen wrote: > > So another idea i am thinking about is doing it like this: > > > > > > > > scripts = [ > > > > "bin/myscript3", # <= backwardscompatible > > > > ( 'bin/', [ 'bin/myscript1', 'bin/myscript2' ] ), > > > > ( 'sbin/', [ 'sbin/mysbinscript1', 'sbin/mysbinscript2' ] ), > > > > ( 'lib/myproject/libexec', [ 'libexec/script' ] ) > > > > ] > > The "scripts" argument is generally considered obsolete - tools like > pip will not generate executable wrappers for scripts specified like > this, which means that they will not work properly on Windows. > > You should really be using (in this case, extending) the setuptools > "console_scripts" entry point. This should be the next step. And it will. The message that scripts from distutils/setuptools is deprecated seems to be a bit unknown out there. http://code.ohloh.net/search?s=setup%20scripts&p=2&pp=0&fl=Python&mp=1&ml=1&me=1&md= 1&ff=1&filterChecked=true So doing both should be the right thing to do. The fact that pip does not support it is imho pips problem. So who will decide if any change to distutils goes through? I have no idea about the python patch process. Any links? Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at michael-jansen.biz Tue Dec 3 10:53:26 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Tue, 03 Dec 2013 10:53:26 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <40770026.o8Us6A34yE@gambit.local.michael-jansen.biz> Message-ID: <3729431.AL3ONb0rYr@gambit.local.michael-jansen.biz> On Tuesday, December 03, 2013 09:37:03 AM Paul Moore wrote: > On 3 December 2013 09:38, Michael Jansen wrote: > > So who will decide if any change to distutils goes through? I have no idea > > about the python patch process. Any links? > > Patches can be posted on http://bugs.python.org. But generally, > changes to distutils are hard to get through so be prepared for some > pushback. I kinda assumed. I can't believe no attempt was made to fix that stuff in the last 10 years or more :) . Found a very old bug. Closed. Perhaps a temporary solution could be to implement both (scripts, console_scripts) for setuptools and THEN try to upstream the parts that make sense for distutils too. Being able to show it works and is backwards compatible could help the case. I guess i will give that a try. -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 3 11:19:08 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 20:19:08 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 19:22, Paul Moore wrote: > On 3 December 2013 08:48, Nick Coghlan wrote: >> This means that one key reason I want to recommend it for the cases where it >> is a good fit (i.e. the scientific Python stack) is so we can explicitly >> advise *against* using it in other cases where it will just add complexity >> without adding value. >> >> Saying nothing is not an option, since people are already confused. Saying >> to never use it isn't an option either, since bootstrapping conda first *is* >> a substantially simpler cross-platform way to get up to date scientific >> Python software on to your system. The alternatives are platform specific >> and (at least in the Linux distro case) slower to get updates. > > But you're not saying "use conda for the scientific Python stack". > You're saying to use it "when you have binary external dependencies" > which is a phrase that I (and I suspect many Windows users) don't > really understand and will take to mean "C extensions, or at least > ones that interface to another library, sich as pyyaml, lxml, ...) That's not what I meant though - I only mean the case where there's a binary dependency that's completely outside the Python ecosystem and can't be linked or bundled because it needs to be shared between multiple components on the Python side. However, there haven't been any compelling examples presented other than the C runtime (which wheel needs to handle as part of the platform tag and/or the ABI tag) and the scientific stack, so I agree limiting the recommendation to the scientific stack is a reasonable approach. Only folks that actually understand the difference between static and dynamic linking and wrapper modules vs self-contained accelerator modules are likely to understand what "shared external binary dependency" means, so I agree it's not a useful phrase to use in a recommendation aimed at folks that aren't already experienced developers. If Windows and Mac OS X users have alternatives they strongly favour over conda that are virtualenv compatible, then sure, we can consider those as well, but I'm not aware of any (as the "virtualenv compatible" bit rules out anything based on platform installers). > Also, this presumes an either/or situation. What about someone who > just wants to use matplotlib to display a graph of some business data? > Is matplotlib part of "the scientific stack"? Should I use conda > *just* to get matplotlib in an otherwise wheel-based application? Ultimately, it depends on if matplotlib is coupled to the NumPy build options or not. However, I think the more practical recommendation would be to say: - if there's no wheel - and you can't build it from source yourself - then you can try "pip install conda && conda init && conda install " as a fallback option. And then we encourage the conda devs to follow the installation database standard properly (if they aren't already), so things installed with conda play nice with things installed with pip. It sounds like we also need to get them to ensure they're using the right compiler/C runtime on Windows so their packages are interoperable with the standard python.org installers. > Or > how about a scientist that wants wxPython (to use Chris' example)? > Apparently the conda repo doesn't include wxPython, so do they need to > learn how to install pip into a conda environment? (Note that there's > no wxPython wheel, so this isn't a good example yet, but I'd hope it > will be in due course...) No, it's the other way around - for cases where wheels aren't yet available, but conda provides it, then we should try to ensure that "pip install conda && conda init && conda install " does the right thing (including conda upgrading previously pip installed packages when necessary, as well as bailing out gracefully when it needs to). At the moment, we're getting people trying to use conda as the base, and stuff falling apart at a later stage, since conda isn't structured properly to handle use cases other than the scientific one where simplicity and repeatabilitly for people that aren't primarily developers trumps platform integration and easier handling of security updates. > Reducing confusion is good, I'm all for that. But we need to have a > clear picture of what we're saying before we can state it clearly... Agreed, that's a large part of why I started this thread. It's definitely clarified several points for me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Dec 3 11:23:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 20:23:11 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 20:19, Nick Coghlan wrote: > Only folks that actually understand the difference between > static and dynamic linking and wrapper modules vs self-contained > accelerator modules are likely to understand what "shared external > binary dependency" means, so I agree it's not a useful phrase to use > in a recommendation aimed at folks that aren't already experienced > developers. "... aren't already experienced C/C++/etc developers". There are lots of higher level languages (including Python itself) that people can be an experienced in and still have never had the "pleasure" of learning the ins and outs of dynamic linking and binary ABIs. Foundations made of sand - it isn't surprising that software sometimes fails, it's a miracle that it ever works at all :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Tue Dec 3 11:36:48 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 3 Dec 2013 10:36:48 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 10:19, Nick Coghlan wrote: >> Or >> how about a scientist that wants wxPython (to use Chris' example)? >> Apparently the conda repo doesn't include wxPython, so do they need to >> learn how to install pip into a conda environment? (Note that there's >> no wxPython wheel, so this isn't a good example yet, but I'd hope it >> will be in due course...) > > No, it's the other way around - for cases where wheels aren't yet > available, but conda provides it, then we should try to ensure that > "pip install conda && conda init && conda install " does the > right thing (including conda upgrading previously pip installed > packages when necessary, as well as bailing out gracefully when it > needs to). Perhaps it would help if there were wheels for conda and its dependencies. "pycosat" (whatever that is) breaks when I pip install conda: $ pip install conda Downloading/unpacking pycosat (from conda) Downloading pycosat-0.6.0.tar.gz (58kB): 58kB downloaded Running setup.py egg_info for package pycosat Downloading/unpacking pyyaml (from conda) Downloading PyYAML-3.10.tar.gz (241kB): 241kB downloaded Running setup.py egg_info for package pyyaml Installing collected packages: pycosat, pyyaml Running setup.py install for pycosat building 'pycosat' extension q:\tools\MinGW\bin\gcc.exe -mdll -O -Wall -Iq:\tools\Python27\include -IQ:\venv\PC -c pycosat.c -o build\temp.win32-2.7\Release\pycosat.o In file included from pycosat.c:18:0: picosat.c: In function 'picosat_stats': picosat.c:8179:4: warning: unknown conversion type character 'l' in format [-Wformat] picosat.c:8179:4: warning: too many arguments for format [-Wformat-extra-args] picosat.c:8180:4: warning: unknown conversion type character 'l' in format [-Wformat] picosat.c:8180:4: warning: too many arguments for format [-Wformat-extra-args] In file included from pycosat.c:18:0: picosat.c: At top level: picosat.c:8210:26: fatal error: sys/resource.h: No such file or directory compilation terminated. error: command 'gcc' failed with exit status 1 Complete output from command Q:\venv\Scripts\python.exe -c "import setuptools;__file__='Q:\\venv\\build\\pycosat\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\docume~1\enojb\locals~1\temp\pip-lobu76-record\install-record.txt --single-version-externally-managed --install-headers Q:\venv\include\site\python2.7: running install running build running build_py creating build creating build\lib.win32-2.7 copying test_pycosat.py -> build\lib.win32-2.7 running build_ext building 'pycosat' extension creating build\temp.win32-2.7 creating build\temp.win32-2.7\Release q:\tools\MinGW\bin\gcc.exe -mdll -O -Wall -Iq:\tools\Python27\include -IQ:\venv\PC -c pycosat.c -o build\temp.win32-2.7\Release\pycosat.o In file included from pycosat.c:18:0: picosat.c: In function 'picosat_stats': picosat.c:8179:4: warning: unknown conversion type character 'l' in format [-Wformat] picosat.c:8179:4: warning: too many arguments for format [-Wformat-extra-args] picosat.c:8180:4: warning: unknown conversion type character 'l' in format [-Wformat] picosat.c:8180:4: warning: too many arguments for format [-Wformat-extra-args] In file included from pycosat.c:18:0: picosat.c: At top level: picosat.c:8210:26: fatal error: sys/resource.h: No such file or directory compilation terminated. error: command 'gcc' failed with exit status 1 ---------------------------------------- Cleaning up... Command Q:\venv\Scripts\python.exe -c "import setuptools;__file__='Q:\\venv\\build\\pycosat\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\docume~1\enojb\locals~1\temp\pip-lobu76-record\install-record.txt --single-version-externally-managed --install-headers Q:\venv\include\site\python2.7 failed with error code 1 in Q:\venv\build\pycosat Storing complete log in c:/Documents and Settings/enojb\pip\pip.log Oscar From pachi at rvburke.com Tue Dec 3 11:38:36 2013 From: pachi at rvburke.com (Pachi) Date: Tue, 03 Dec 2013 11:38:36 +0100 Subject: [Distutils] Handling the binary dependency management problem - wording In-Reply-To: References: Message-ID: <529DB4AC.8010507@rvburke.com> El 03/12/2013 10:22, Paul Moore escribi?: > On 3 December 2013 08:48, Nick Coghlan wrote: >> This means that one key reason I want to recommend it for the cases where it >> is a good fit (i.e. the scientific Python stack) is so we can explicitly >> advise *against* using it in other cases where it will just add complexity >> without adding value. >> >> Saying nothing is not an option, since people are already confused. Saying >> to never use it isn't an option either, since bootstrapping conda first *is* >> a substantially simpler cross-platform way to get up to date scientific >> Python software on to your system. The alternatives are platform specific >> and (at least in the Linux distro case) slower to get updates. > But you're not saying "use conda for the scientific Python stack". > You're saying to use it "when you have binary external dependencies" > which is a phrase that I (and I suspect many Windows users) don't > really understand and will take to mean "C extensions, or at least > ones that interface to another library, sich as pyyaml, lxml, ...) > > Also, this presumes an either/or situation. What about someone who > just wants to use matplotlib to display a graph of some business data? > Is matplotlib part of "the scientific stack"? Should I use conda > *just* to get matplotlib in an otherwise wheel-based application? Or > how about a scientist that wants wxPython (to use Chris' example)? > Apparently the conda repo doesn't include wxPython, so do they need to > learn how to install pip into a conda environment? (Note that there's > no wxPython wheel, so this isn't a good example yet, but I'd hope it > will be in due course...) > > Reducing confusion is good, I'm all for that. But we need to have a > clear picture of what we're saying before we can state it clearly... > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig A first and non-native try to get a clearer wording for this: "Some collections of python packages may have further compatibility needs than those expressed by the current set of platform tags used in wheels. That is the case of the Python scientific stack, where interoperability depends on the choice of a shared binary data format that is decided at build time. This problem can be solved by packagers' consensus on a common choice of compatibility options or by using curated indices. Also, package managers like conda do additional checks to ensure a coherent set of Python and non-Python packages and may offer at this time a better user experience for package collections with such complex dependencies." Regards, -- Pachi From p.f.moore at gmail.com Tue Dec 3 12:17:29 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 11:17:29 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 10:36, Oscar Benjamin wrote: > Perhaps it would help if there were wheels for conda and its > dependencies. That may well be a good idea. One thing pip does is go to great lengths to *not* have any dependencies (by vendoring everything it needs, and relying only on pure Python code). It looks like the conda devs haven't (yet? ;-)) found the need to do that. So a suitable set of wheels would go a long way to improving the bootstrap experience. Having to have MSVC (or gcc, I guess, if they can get your build issues fixed) if you want to bootstrap conda is a pretty significant roadblock... Paul From ncoghlan at gmail.com Tue Dec 3 12:12:20 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 21:12:20 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 19:11, Paul Moore wrote: > On 3 December 2013 08:48, Nick Coghlan wrote: >>> And wouldn't it be better to make wheel a bit more robust in this regard >>> than add yet another recommended tool to the mix? >> >> Software that works today is generally more useful to end users than >> software that might possibly handle their use case at some currently >> unspecified point in the future :) > > See my experience with conda under Windows. While I'm not saying that > conda "doesn't work", being directed to software that turns out to > have its own set of bugs, different to the ones you're used to, is a > pretty frustrating experience. Yeah, I hit the one where it tries to upgrade the symlinked Python in a virtualenv on POSIX systems and fails: https://github.com/ContinuumIO/conda/issues/360 > (BTW, I raised a bug report. For anyone else that is curious: https://github.com/ContinuumIO/conda/issues/396 In looking for a clear explanation of the runtime compatibility requirements for extensions, I realised that such a thing doesn't appear to exist. And then I realised I wasn't aware of the existence of *any* good overview of C extensions for Python, their benefits, their limitations, alternatives to creating them by hand, and that such a thing might be a good addition to the "Advanced topics" section of the packaging user guide: https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-section-that-covers-binary > Let's see > what the response is like...) Since venv in Python 3.4 has a working --copies option, I bashed away at the conda+venv combination a bit more, and filed another couple of conda bugs: Gets shebang lines wrong in a virtual environment: https://github.com/ContinuumIO/conda/issues/397 Doesn't currently support "python -m conda": https://github.com/ContinuumIO/conda/issues/398 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Tue Dec 3 12:22:52 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 3 Dec 2013 11:22:52 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 1 December 2013 04:15, Nick Coghlan wrote: > > conda has its own binary distribution format, using hash based > dependencies. It's this mechanism which allows it to provide reliable > cross platform binary dependency management, but it's also the same > mechanism that prevents low impact security updates and > interoperability with platform provided packages. Nick can you provide a link to somewhere that explains the hash based dependency thing please? I've read the following... http://docs.continuum.io/conda/ https://speakerdeck.com/teoliphant/packaging-and-deployment-with-conda http://docs.continuum.io/anaconda/index.html http://continuum.io/blog/new-advances-in-conda http://continuum.io/blog/conda http://docs.continuum.io/conda/build.html ...but I see no reference to hash-based dependencies. In fact the only place I have seen a reference to hash-based dependencies is your comment at the bottom of this github issue: https://github.com/ContinuumIO/conda/issues/292 AFAICT conda/binstar are alternatives for pip/PyPI that happen to host binaries for some packages that don't have binaries on PyPI. (conda also provides a different - incompatible - take on virtualenvs but that's not relevant to this proposal). Oscar From ncoghlan at gmail.com Tue Dec 3 12:24:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 21:24:19 +1000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> <40770026.o8Us6A34yE@gambit.local.michael-jansen.biz> Message-ID: On 3 December 2013 19:37, Paul Moore wrote: > On 3 December 2013 09:38, Michael Jansen wrote: >> So who will decide if any change to distutils goes through? I have no idea >> about the python patch process. Any links? > > Patches can be posted on http://bugs.python.org. But generally, > changes to distutils are hard to get through so be prepared for some > pushback. Changes to distutils itself are fairly pointless, since the earliest possible date for publication of any such changes is now as part of Python 3.5 in 2015. The sheer impracticality of that approach when plenty of people are still running Python 2.6 is why we're going through the somewhat painful exercise of decoupling the preferred build system from the standard library :) So, at this point, the appropriate place to make such a change is in setuptools: https://bitbucket.org/pypa/setuptools That will allow it to be made in a way that supports Python 2.6+, whereas a distutils change won't really help anyone. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From theller at ctypes.org Tue Dec 3 12:27:34 2013 From: theller at ctypes.org (Thomas Heller) Date: Tue, 03 Dec 2013 12:27:34 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: Am 03.12.2013 09:56, schrieb Paul Moore: > > The "scripts" argument is generally considered obsolete - tools like > pip will not generate executable wrappers for scripts specified like > this, which means that they will not work properly on Windows. > The Python documentation does not mention this - neither 2.7 nor 3.x. > You should really be using (in this case, extending) the setuptools > "console_scripts" entry point. More and more I get the impression that distributing (writing the setup script and deciding on package formats) Python libraries is currently nearly impossible for the mortal user. distutils, setuptools, wheels... sigh. Thomas From info at michael-jansen.biz Tue Dec 3 12:33:22 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Tue, 03 Dec 2013 12:33:22 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> Message-ID: <2700087.3DAFl4lBSb@gambit.local.michael-jansen.biz> > Changes to distutils itself are fairly pointless, since the earliest > possible date for publication of any such changes is now as part of > Python 3.5 in 2015. The sheer impracticality of that approach when > plenty of people are still running Python 2.6 is why we're going > through the somewhat painful exercise of decoupling the preferred > build system from the standard library :) > > So, at this point, the appropriate place to make such a change is in > setuptools: https://bitbucket.org/pypa/setuptools > > That will allow it to be made in a way that supports Python 2.6+, > whereas a distutils change won't really help anyone. A point well made :) . Will do that. Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Dec 3 12:42:37 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 11:42:37 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: On 3 December 2013 11:27, Thomas Heller wrote: > Am 03.12.2013 09:56, schrieb Paul Moore: > >> >> The "scripts" argument is generally considered obsolete - tools like >> pip will not generate executable wrappers for scripts specified like >> this, which means that they will not work properly on Windows. >> > > The Python documentation does not mention this - neither 2.7 nor 3.x. Fair point. It's more that "scripts" offers you absolutely no help in writing cross-platform scripts, or wrapping scripts in exes (which, although annoying, is necessary on Windows to avoid subtle and frustrating issues), or otherwise doing anything beyond dumping a few files somewhere that will end up on PATH. The entry point mechanism introduced in setuptools, and extended/enhanced as part of the pip/wheel work, offers a much more robust and flexible solution. >> You should really be using (in this case, extending) the setuptools >> "console_scripts" entry point. > > More and more I get the impression that distributing (writing the setup > script and deciding on package formats) Python libraries is currently > nearly impossible for the mortal user. > > distutils, setuptools, wheels... sigh. I feel your pain. There is work ongoing to write a "packaging user guide" which documents best practice and sticks to providing a "one way to do things" approach. But it's very much a work in progress at this point, and things are changing rapidly right now, so it's some way away from being useful yet (don't look at the section on packaging your project unless you are fond of reading sections that say "FIXME" - or you want to help write those sections :-)) Even as someone directly involved in pip/wheel development, I struggle every time I need to write a setup.py. All I can say is that we're trying to cut through the confusion as fast as we can... Paul From ncoghlan at gmail.com Tue Dec 3 12:54:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 21:54:02 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 21:22, Oscar Benjamin wrote: > AFAICT conda/binstar are alternatives for pip/PyPI that happen to host > binaries for some packages that don't have binaries on PyPI. (conda > also provides a different - incompatible - take on virtualenvs but > that's not relevant to this proposal). It sounds like I may have been confusing two presentations at the packaging mini-summit, as I would have sworn conda used hashes to guarantee a consistent set of packages. I know I have mixed up features between hashdist and conda in the past (and there have been some NixOS features mixed in there as well), so it wouldn't be the first time that has happened - the downside of mining different distribution systems for ideas is that sometimes I forget where I encountered particular features :) If conda doesn't offer such an internal consistency guarantee for published package sets, then I agree with the criticism that it's just an alternative to running a private PyPI index server hosting wheel files pre-built with particular options, and thus it becomes substantially less interesting to me :( Under that model, what conda is doing is *already covered* in the draft metadata 2.0 spec (as of the changes I posted about the other day), since that now includes an "integrator suffix" (to indicate when a downstream rebuilder has patched the software), as well as a "python.integrator" metadata extension to give details of the rebuild. The namespacing in the wheel case is handled by not allowing rebuilds to be published on PyPI - they have to be published on a separate index server, and thus can be controlled based on where you tell pip to look. So, I apologise for starting the thread based on what appears to be a fundamentally false premise, although I think it has still been useful despite that error on my part (as the user confusion is real, even though my specific proposal no longer seems as useful as I first thought). I believe helping the conda devs to get it to play nice with virtual environments is still a worthwhile exercise though (even if just by pointing out areas where it *doesn't* currently interoperate well, as we've been doing in the last day or so), and if the conda bootstrapping issue is fixed by publishing wheels (or vendoring dependencies), then "try conda if there's no wheel" may still be a reasonable fallback recommendation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Tue Dec 3 13:49:27 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 3 Dec 2013 12:49:27 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 11:54, Nick Coghlan wrote: > On 3 December 2013 21:22, Oscar Benjamin wrote: >> AFAICT conda/binstar are alternatives for pip/PyPI that happen to host >> binaries for some packages that don't have binaries on PyPI. (conda >> also provides a different - incompatible - take on virtualenvs but >> that's not relevant to this proposal). > > It sounds like I may have been confusing two presentations at the > packaging mini-summit, as I would have sworn conda used hashes to > guarantee a consistent set of packages. I know I have mixed up > features between hashdist and conda in the past (and there have been > some NixOS features mixed in there as well), so it wouldn't be the > first time that has happened - the downside of mining different > distribution systems for ideas is that sometimes I forget where I > encountered particular features :) I had the same confusion with hashdist at the start of this thread when I said that conda was targeted at HPC. So if we both make the same mistake I guess it's forgiveable :) > If conda doesn't offer such an internal consistency guarantee for > published package sets, then I agree with the criticism that it's just > an alternative to running a private PyPI index server hosting wheel > files pre-built with particular options, and thus it becomes > substantially less interesting to me :( Perhaps Travis who is still CC'ed here could comment on this since it is apparent that no one here really understands what conda is and he apparently works for Continuum Analytics so should (hopefully) know a little more... > Under that model, what conda is doing is *already covered* in the > draft metadata 2.0 spec (as of the changes I posted about the other > day), since that now includes an "integrator suffix" (to indicate when > a downstream rebuilder has patched the software), as well as a > "python.integrator" metadata extension to give details of the rebuild. > The namespacing in the wheel case is handled by not allowing rebuilds > to be published on PyPI - they have to be published on a separate > index server, and thus can be controlled based on where you tell pip > to look. Do you mean to say that PyPI can (should) only host a binary-compatible set of wheels and that other index servers should do the same? I still think that there needs to be some kind of compatibility tags either way. > So, I apologise for starting the thread based on what appears to be a > fundamentally false premise, although I think it has still been useful > despite that error on my part (as the user confusion is real, even > though my specific proposal no longer seems as useful as I first > thought). > > I believe helping the conda devs to get it to play nice with virtual > environments is still a worthwhile exercise though (even if just by > pointing out areas where it *doesn't* currently interoperate well, as > we've been doing in the last day or so), and if the conda > bootstrapping issue is fixed by publishing wheels (or vendoring > dependencies), then "try conda if there's no wheel" may still be a > reasonable fallback recommendation. Well for a start conda (at least according to my failed build) over-writes the virtualenv activate scripts with its own scripts that do something completely different and can't even be called with the same signature. So it looks to me as if there is no intention of virtualenv compatibility. As for "try conda if there's no wheel" according to what I've read that seems to be what people who currently use conda do. I thought about another thing during the course of this thread. To what extent can Provides/Requires help out with the binary incompatibility problems? For example numpy really does provide multiple interfaces: 1) An importable Python module that can be used from Python code. 2) A C-API that can be used by compiled C-extensions. 3) BLAS/LAPACK libraries with a particular Fortran ABI to any other libraries in the same process. Perhaps the solution is that a build of a numpy wheel should clarify explicitly what it Provides at each level e.g.: Provides: numpy Provides: numpy-capi-v1 Provides: numpy-openblas-g77 Then a built wheel for scipy can Require the same things. Cristoph Gohlke could provide a numpy wheel with: Provides: numpy Provides: numpy-capi-v1 Provides: numpy-intelmkl And his scipy wheel can require the same. This would mean that pip would understand the binary dependency problems during dependency resolution and could reject an incompatible wheel at install time as well as being able to find a compatible wheel automatically if one exists in the server. Unlike the hash-based dependencies we can see that it is possible to depend on the numpy C-API without necessarily depending on any particular BLAS/LAPACK library and Fortran compiler combination. The confusing part would be that then a built wheel doesn't Provide the same thing as the corresponding sdist. How would anyone know what would be Provided by an sdist without first building it into a wheel? Would there need to be a way for pip to tell the sdist what pip wants it to Provide when building it? Oscar From ncoghlan at gmail.com Tue Dec 3 13:28:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 22:28:21 +1000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: On 3 December 2013 21:42, Paul Moore wrote: > On 3 December 2013 11:27, Thomas Heller wrote: >> More and more I get the impression that distributing (writing the setup >> script and deciding on package formats) Python libraries is currently >> nearly impossible for the mortal user. >> >> distutils, setuptools, wheels... sigh. > > I feel your pain. There is work ongoing to write a "packaging user > guide" which documents best practice and sticks to providing a "one > way to do things" approach. But it's very much a work in progress at > this point, and things are changing rapidly right now, so it's some > way away from being useful yet (don't look at the section on packaging > your project unless you are fond of reading sections that say "FIXME" > - or you want to help write those sections :-)) > > Even as someone directly involved in pip/wheel development, I struggle > every time I need to write a setup.py. All I can say is that we're > trying to cut through the confusion as fast as we can... Aye, we're suffering from the fact that: - the distutils docs haven't been substantially updated in years (remember folks, you need to make sure to build your CPython extensions with Visual C++ 7.1!) - the setuptools docs are improving, but still have large elements of assuming you already know how to use distutils - the distribute and distutils2 docs assumed a future that never eventuated We've also been focusing our efforts on getting the installer side of things sorted out moreso than the build side, and PEP 453 is really the culmination of that effort (mostly implemented now, as there's just a few more issues for pip 1.5 final, along with the docs update to go on the CPython side. See http://bugs.python.org/issue19347 for details, since I can't apply the PEP453 tag on the pip side of things). Metadata 2.0 is then about clearly defining the full data model we want to support for distribution, and I think that will help us better document the build side of things. Getting the packaging user guide up to an at least somewhat usable state is highly desirable though, since that's what I plan to reference from the Python 3.4 docs. Even if it's incomplete, it would be difficult for it to be more misleading than the current distutils docs that are shipping with CPython :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Dec 3 14:53:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 3 Dec 2013 23:53:21 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 22:49, Oscar Benjamin wrote: > On 3 December 2013 11:54, Nick Coghlan wrote: >> I believe helping the conda devs to get it to play nice with virtual >> environments is still a worthwhile exercise though (even if just by >> pointing out areas where it *doesn't* currently interoperate well, as >> we've been doing in the last day or so), and if the conda >> bootstrapping issue is fixed by publishing wheels (or vendoring >> dependencies), then "try conda if there's no wheel" may still be a >> reasonable fallback recommendation. > > Well for a start conda (at least according to my failed build) > over-writes the virtualenv activate scripts with its own scripts that > do something completely different and can't even be called with the > same signature. So it looks to me as if there is no intention of > virtualenv compatibility. Historically there hadn't been much work in that direction, but I think there's been some increasing awareness of the importance of compatibility with the standard tools recently (I'm not certain, but the acceptance of PEP 453 may have had some impact there). I also consider Travis a friend, and have bent his ear over some of the compatibility issues, as well as the fact that pip has to handle additional usage scenarios that just aren't relevant to most of the scientific community, but are critical for professional application developers and system integrators :) The recent addition of "conda init" (in order to reuse a venv or virtualenv environment) was a big step in the right direction, and there's an issue filed about activate getting clobbered: https://github.com/ContinuumIO/conda/issues/374 (before conda init, you couldn't really mix conda and virtualenv, so the fact they both had activate scripts didn't matter. Now it does, since it affects the usability of conda init) > As for "try conda if there's no wheel" according to what I've read > that seems to be what people who currently use conda do. > > I thought about another thing during the course of this thread. To > what extent can Provides/Requires help out with the binary > incompatibility problems? For example numpy really does provide > multiple interfaces: > 1) An importable Python module that can be used from Python code. > 2) A C-API that can be used by compiled C-extensions. > 3) BLAS/LAPACK libraries with a particular Fortran ABI to any other > libraries in the same process. > > Perhaps the solution is that a build of a numpy wheel should clarify > explicitly what it Provides at each level e.g.: > > Provides: numpy > Provides: numpy-capi-v1 > Provides: numpy-openblas-g77 > > Then a built wheel for scipy can Require the same things. Cristoph > Gohlke could provide a numpy wheel with: > > Provides: numpy > Provides: numpy-capi-v1 > Provides: numpy-intelmkl Hmm, I likely wouldn't build it into the core requirement system (that all operates at the distribution level), but the latest metadata updates split out a bunch of the optional stuff to extensions (see https://bitbucket.org/pypa/pypi-metadata-formats/src/default/standard-metadata-extensions.rst). What we're really after at this point is the ability to *detect* conflicts if somebody tries to install incompatible builds into the same virtual environment (e.g. you installed from custom index server originally, but later you forget and install from PyPI). So perhaps we could have a "python.expects" extension, where we can assert certain things about the metadata of other distributions in the environment. So, say that numpy were to define a custom extension where they can define the exported binary interfaces: "extensions": { "numpy.compatibility": { "api_version": 1, "fortran_abi": "openblas-g77" } } And for the Gohlke rebuilds: "extensions": { "numpy.compatibility": { "api_version": 1, "fortran_abi": "intelmki" } } Then another component might have in its metadata: "extensions": { "python.expects": { "numpy": { "extensions": { "numpy.compatibility": { "fortran_abi": "openblas-g77" } } } } } The above would be read as '"this distribution expects the numpy distribution in this environment to publish the "numpy.compatibility" extension in its metadata, with the "fortran_abi" field set to "openblas-g77"' If you attempted to install that component into an environment with the intelmki FORTRAN ABI declared, it would fail, since the expectation wouldn't match the reality. > And his scipy wheel can require the same. This would mean that pip > would understand the binary dependency problems during dependency > resolution and could reject an incompatible wheel at install time as > well as being able to find a compatible wheel automatically if one > exists in the server. Unlike the hash-based dependencies we can see > that it is possible to depend on the numpy C-API without necessarily > depending on any particular BLAS/LAPACK library and Fortran compiler > combination. I like the general idea of being able to detect conflicts through the published metadata, but would like to use the extension mechanism to avoid name conflicts. > The confusing part would be that then a built wheel doesn't Provide > the same thing as the corresponding sdist. How would anyone know what > would be Provided by an sdist without first building it into a wheel? > Would there need to be a way for pip to tell the sdist what pip wants > it to Provide when building it? I think that's a separate (harder) problem, but one that the expectation approach potentially solves, since we'd just have to provide a list of expectations for a distribution to the build process (and individual distributions would have full control over defining what expectations will influence the build process, most likely through custom extensions). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Tue Dec 3 16:30:53 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Tue, 3 Dec 2013 15:30:53 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 13:53, Nick Coghlan wrote: > On 3 December 2013 22:49, Oscar Benjamin wrote: > > Hmm, I likely wouldn't build it into the core requirement system (that > all operates at the distribution level), but the latest metadata > updates split out a bunch of the optional stuff to extensions (see > https://bitbucket.org/pypa/pypi-metadata-formats/src/default/standard-metadata-extensions.rst). > What we're really after at this point is the ability to *detect* > conflicts if somebody tries to install incompatible builds into the > same virtual environment (e.g. you installed from custom index server > originally, but later you forget and install from PyPI). > > So perhaps we could have a "python.expects" extension, where we can > assert certain things about the metadata of other distributions in the > environment. So, say that numpy were to define a custom extension > where they can define the exported binary interfaces: > > "extensions": { > "numpy.compatibility": { > "api_version": 1, > "fortran_abi": "openblas-g77" > } > } [snip] > > I like the general idea of being able to detect conflicts through the > published metadata, but would like to use the extension mechanism to > avoid name conflicts. Helping to prevent borken installs in this way would definitely be an improvement. It would be a real shame though if PyPI would contain all the metadata needed to match up compatible binary wheels but pip would only use it to show error messages rather than to actually locate the wheel that the user wants. Oscar From qwcode at gmail.com Tue Dec 3 20:54:07 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 3 Dec 2013 11:54:07 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > If conda doesn't offer such an internal consistency guarantee for > published package sets, then I agree with the criticism that it's just > an alternative to running a private PyPI index server hosting wheel > files pre-built with particular options, and thus it becomes > substantially less interesting to me :( > well, except that the anaconda index covers non-python projects like "qt", which a private wheel index wouldn't cover (at least with the normal intended use of wheels) -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Tue Dec 3 21:22:13 2013 From: theller at ctypes.org (Thomas Heller) Date: Tue, 03 Dec 2013 21:22:13 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <4901190.YcaFg1U7BQ@gambit.local.michael-jansen.biz> Message-ID: Am 03.12.2013 09:56, schrieb Paul Moore: > The "scripts" argument is generally considered obsolete - tools like > pip will not generate executable wrappers for scripts specified like > this, which means that they will not work properly on Windows. Maybe pip (or bdist_wheel ?) should emit a Warning or an error, if it finds a setup-script with "scripts" argument? Thomas From ncoghlan at gmail.com Tue Dec 3 21:31:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Dec 2013 06:31:38 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 Dec 2013 05:54, "Marcus Smith" wrote: > > >> >> If conda doesn't offer such an internal consistency guarantee for >> published package sets, then I agree with the criticism that it's just >> an alternative to running a private PyPI index server hosting wheel >> files pre-built with particular options, and thus it becomes >> substantially less interesting to me :( > > > well, except that the anaconda index covers non-python projects like "qt", which a private wheel index wouldn't cover (at least with the normal intended use of wheels) Ah, true - there's still the non-trivial matter of getting hold of the external dependencies *themselves*. Anyway, this thread has at least satisfied me that we don't need to rush anything at this point - we can see how the conda folks go handling the interoperability issues, come up with an overview of the situation for creating and publishing binary extensions, keep working on getting the Python 3.4 + pip 1.5 combination out the door, and then decide later exactly how we think conda fits into the overall picture, as well as what influence the problems it solves for the scientific stack should have on the metadata 2.0 design. Cheers, Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 3 21:40:46 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Dec 2013 06:40:46 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 Dec 2013 01:31, "Oscar Benjamin" wrote: > > On 3 December 2013 13:53, Nick Coghlan wrote: > > On 3 December 2013 22:49, Oscar Benjamin wrote: > > > > Hmm, I likely wouldn't build it into the core requirement system (that > > all operates at the distribution level), but the latest metadata > > updates split out a bunch of the optional stuff to extensions (see > > https://bitbucket.org/pypa/pypi-metadata-formats/src/default/standard-metadata-extensions.rst ). > > What we're really after at this point is the ability to *detect* > > conflicts if somebody tries to install incompatible builds into the > > same virtual environment (e.g. you installed from custom index server > > originally, but later you forget and install from PyPI). > > > > So perhaps we could have a "python.expects" extension, where we can > > assert certain things about the metadata of other distributions in the > > environment. So, say that numpy were to define a custom extension > > where they can define the exported binary interfaces: > > > > "extensions": { > > "numpy.compatibility": { > > "api_version": 1, > > "fortran_abi": "openblas-g77" > > } > > } > [snip] > > > > I like the general idea of being able to detect conflicts through the > > published metadata, but would like to use the extension mechanism to > > avoid name conflicts. > > Helping to prevent borken installs in this way would definitely be an > improvement. It would be a real shame though if PyPI would contain all > the metadata needed to match up compatible binary wheels but pip would > only use it to show error messages rather than to actually locate the > wheel that the user wants. One of the side benefits of the TUF end-to-end security proposal is that it allows for more reliable local metadata caching. That means that even if a mechanism like this was initially only used for conflict detection, it could eventually also be used to choose between multiple possible resolutions of a dependency. Tangentially related, I'm considering adding an additional recommended URL field to the "python.integrator" extension, which would be a pointer to a custom index server operated by the integrator. Cheers, Nick. > > > Oscar -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Dec 3 01:00:55 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 2 Dec 2013 16:00:55 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: Side note about naming: I'm no expert, but I'm pretty sure "Anoconda" is a python distribution -- python itself and set of pre-build packages. "conda" is the package manager that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is an open-source project, and thus could be used by any of us completely apart from the Anoconda distribution. On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore wrote: > > had to resort to Google to try to figure out what dev libraries I needed. > > But that's a *build* issue, surely? How does that relate to installing > Nikola from a set of binary wheels? > > Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of users that want binaries, and want them to "just work" -- we've had mpkg packages for a good while, analogous to Windows installers. Binary eggs never worked quite right, 'cause setuptools didn't understand "universal" binaries -- but it wasn't that far from working. Not really tested much yet, but it ;looks like binary wheels should be just fine. The concern there is that someone will be running, say, a homebrew-built python, and accidentally install a binary wheel built for the python.org python -- we should address that with better platform tags (and making sure pip at least give a warning if you try to install an incompatible wheel...) So what problem are we trying to solve here? 1) It's still a pain to actually build the packages -- similarly to Windows, you really need to build the dependent libraries statically and link them in - and you need to make sure that you build them with teh right sdk, and universally -- this is hard to do right. - does Conda help you do any of that??? 2) non-python binary dependencies: As it turns out, a number of python packages depend on the same third-party non-python dependencies: I have quite a few that use libpng, libfreetype, libhdf, ??? currently if you want to distribute binary python packages, you need to statically link or supply the dlls, so we end up with multiple coples of the same lib -- is this a problem? Maybe not -- memory is pretty cheap these days, and maybe different packages actually rely on different versions of the dependencies -- this way, at least the package builder controls that. Anoconda (the distribution seems to address this by having conda packages that are essentially containers for the shared libs, and other packages that need those libs depend on them. I like this method, but it seems to me to be more a feature of the Anoconda distribution than the conda package manager -- in fact, I've been thinking of doing this exact same thing with binary wheels -- I haven't tried it yet, but don't see why it wouldn't work. I understand you are thinking about non-Python libraries, but all I > can say is that this has *never* been an issue to my knowledge in the > Windows world. yes, it's a HUGE issue in the Windows world -- in fact such a huge issue that almost non one ever tries to build things themselves, or build a different python distro -- so, in fact, when someone does make a binary, it's pretty likely to work. But those binaries are a major pain to build! (by the way, over on python-dev there has been a recent discussion about stackless building a new python2.7 windows binary with a newer MS compiler -- which will then create exacty these issues...) > Outside the scientific space, crypto libraries are also notoriously hard > to > > build, as are game engines and GUI toolkits. (I guess database bindings > > could also be a problem in some cases) > > Build issues again... Yes, major ones. (another side note: you can't get wxPython for OS-X to work with Anoconda -- there is no conda binary package, and python itself is not built in a way that it can access the window manager ... so no, this stuff in NOT suddenly easier with conda.) Again, can we please be clear here? On Windows, there is no issue that > I am aware of. Wheels solve the binary distribution issue fine in that > environment They will if/when we make sure that the wheel contains meta-data about what compiler (really run-time version) was used for the python build and wheel build -- but we should, indeed, do that. > This is why I suspect there will be a better near term effort/reward > > trade-off in helping the conda folks improve the usability of their > platform > > than there is in trying to expand the wheel format to cover arbitrary > binary > > dependencies. > and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on what conda offers that isn't quite easy to address with binary wheels. And it seems to need help too, so it will play better with virtualenv.... If conda really is a better solution, then I suppose we could go deprecate wheel before it gets too much "traction"...;-) But let's please not another one to the mix to confuse people. Excuse me if I'm feeling a bit negative towards this announcement. > I've spent many months working on, and promoting, the wheel + pip > solution, to the point where it is now part of Python 3.4. And I was really lookingn forward to it as a solution for OS-X too.... I'm hoping I've misunderstood here. Please clarify. Preferably with > specifics for Windows (as "conda is a known stable platform" simply > isn't true for me...) - I accept you're not a Windows user, so a > pointer to already-existing documentation is fine (I couldn't find any > myself). > I appreciate the Windows viewpoint -- and I think we really do want one solution for all. The linux distros already do all this let them keep doing it.... -Chris PS: a number of scipy-related packages have been promoting Anoconda and Canopy as a way to get their package without dealing with building (rather than say, providing binary wheels). That works great for the SciPy Stack, but has NOT worked well for others. My example: I'm teaching an Intro to Python class -- I really like iPython, so have been using it for demos and recommend it to Students. The IPython web site makes it look like you really need to go get Anaconda or Canopy if you want iPython -- so a number of my students went and did that. All well. Until I did the extra class on wxPython, and now I've got a bunch of students that have no way to install it. And they mostly had no idea that they were running a different Python at all... Anyway -- this is a mess, particularly on OS-X (now we have python binaries from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???) So yes, we need a solution, but I think binary wheels are a pretty good one, and I'm not sure what conda buys us... -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Dec 3 22:13:37 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Dec 2013 16:13:37 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: I think Wheels are the way forward for Python dependencies. Perhaps not for things like fortran. I hope that the scientific community can start publishing wheels at least in addition too. I don't believe that Conda will gain the mindshare that pip has outside of the scientific community so I hope we don't end up with two systems that can't interoperate. > On Dec 2, 2013, at 7:00 PM, Chris Barker wrote: > > Side note about naming: > > I'm no expert, but I'm pretty sure "Anoconda" is a python distribution -- python itself and set of pre-build packages. "conda" is the package manager that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is an open-source project, and thus could be used by any of us completely apart from the Anoconda distribution. > > > On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore wrote: >> > had to resort to Google to try to figure out what dev libraries I needed. >> >> But that's a *build* issue, surely? How does that relate to installing >> Nikola from a set of binary wheels? > Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of users that want binaries, and want them to "just work" -- we've had mpkg packages for a good while, analogous to Windows installers. Binary eggs never worked quite right, 'cause setuptools didn't understand "universal" binaries -- but it wasn't that far from working. Not really tested much yet, but it ;looks like binary wheels should be just fine. The concern there is that someone will be running, say, a homebrew-built python, and accidentally install a binary wheel built for the python.org python -- we should address that with better platform tags (and making sure pip at least give a warning if you try to install an incompatible wheel...) > > So what problem are we trying to solve here? > > 1) It's still a pain to actually build the packages -- similarly to Windows, you really need to build the dependent libraries statically and link them in - and you need to make sure that you build them with teh right sdk, and universally -- this is hard to do right. > - does Conda help you do any of that??? > > 2) non-python binary dependencies: As it turns out, a number of python packages depend on the same third-party non-python dependencies: I have quite a few that use libpng, libfreetype, libhdf, ??? currently if you want to distribute binary python packages, you need to statically link or supply the dlls, so we end up with multiple coples of the same lib -- is this a problem? Maybe not -- memory is pretty cheap these days, and maybe different packages actually rely on different versions of the dependencies -- this way, at least the package builder controls that. > > Anoconda (the distribution seems to address this by having conda packages that are essentially containers for the shared libs, and other packages that need those libs depend on them. I like this method, but it seems to me to be more a feature of the Anoconda distribution than the conda package manager -- in fact, I've been thinking of doing this exact same thing with binary wheels -- I haven't tried it yet, but don't see why it wouldn't work. > >> I understand you are thinking about non-Python libraries, but all I >> can say is that this has *never* been an issue to my knowledge in the >> Windows world. > > yes, it's a HUGE issue in the Windows world -- in fact such a huge issue that almost non one ever tries to build things themselves, or build a different python distro -- so, in fact, when someone does make a binary, it's pretty likely to work. But those binaries are a major pain to build! > > (by the way, over on python-dev there has been a recent discussion about stackless building a new python2.7 windows binary with a newer MS compiler -- which will then create exacty these issues...) > >> > Outside the scientific space, crypto libraries are also notoriously hard to >> > build, as are game engines and GUI toolkits. (I guess database bindings >> > could also be a problem in some cases) >> >> Build issues again... > > Yes, major ones. > > (another side note: you can't get wxPython for OS-X to work with Anoconda -- there is no conda binary package, and python itself is not built in a way that it can access the window manager ... so no, this stuff in NOT suddenly easier with conda.) > >> Again, can we please be clear here? On Windows, there is no issue that >> I am aware of. Wheels solve the binary distribution issue fine in that >> environment > > They will if/when we make sure that the wheel contains meta-data about what compiler (really run-time version) was used for the python build and wheel build -- but we should, indeed, do that. > >> > This is why I suspect there will be a better near term effort/reward >> > trade-off in helping the conda folks improve the usability of their platform >> > than there is in trying to expand the wheel format to cover arbitrary binary >> > dependencies. > > and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on what conda offers that isn't quite easy to address with binary wheels. And it seems to need help too, so it will play better with virtualenv.... > > If conda really is a better solution, then I suppose we could go deprecate wheel before it gets too much "traction"...;-) But let's please not another one to the mix to confuse people. > >> Excuse me if I'm feeling a bit negative towards this announcement. >> I've spent many months working on, and promoting, the wheel + pip >> solution, to the point where it is now part of Python 3.4. > > And I was really lookingn forward to it as a solution for OS-X too.... > >> I'm hoping I've misunderstood here. Please clarify. Preferably with >> specifics for Windows (as "conda is a known stable platform" simply >> isn't true for me...) - I accept you're not a Windows user, so a >> pointer to already-existing documentation is fine (I couldn't find any >> myself). > > I appreciate the Windows viewpoint -- and I think we really do want one solution for all. > > The linux distros already do all this let them keep doing it.... > > -Chris > > PS: a number of scipy-related packages have been promoting Anoconda and Canopy as a way to get their package without dealing with building (rather than say, providing binary wheels). That works great for the SciPy Stack, but has NOT worked well for others. My example: > > I'm teaching an Intro to Python class -- I really like iPython, so have been using it for demos and recommend it to Students. The IPython web site makes it look like you really need to go get Anaconda or Canopy if you want iPython -- so a number of my students went and did that. All well. Until I did the extra class on wxPython, and now I've got a bunch of students that have no way to install it. And they mostly had no idea that they were running a different Python at all... > > Anyway -- this is a mess, particularly on OS-X (now we have python binaries from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???) So yes, we need a solution, but I think binary wheels are a pretty good one, and I'm not sure what conda buys us... > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Dec 3 22:34:04 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 3 Dec 2013 13:34:04 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > Anoconda (the distribution seems to address this by having conda packages > that are essentially containers for the shared libs, and other packages > that need those libs depend on them. I like this method, but it seems to me > to be more a feature of the Anoconda distribution than the conda package > manager -- in fact, I've been thinking of doing this exact same thing with > binary wheels -- I haven't tried it yet, but don't see why it wouldn't work. > > 3 or 4 us now have mentioned curiosity in converting anaconda packages to wheels (with specific interest in the non-python lib dependencies as wheels). Anyone who tries this, please post your success or lack thereof. I'm pretty curious. > The IPython web site makes it look like you really need to go get Anaconda > or Canopy if you want iPython > > interesting... -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Dec 3 22:44:01 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 21:44:01 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 21:34, Marcus Smith wrote: >> Anoconda (the distribution seems to address this by having conda packages >> that are essentially containers for the shared libs, and other packages that >> need those libs depend on them. I like this method, but it seems to me to be >> more a feature of the Anoconda distribution than the conda package manager >> -- in fact, I've been thinking of doing this exact same thing with binary >> wheels -- I haven't tried it yet, but don't see why it wouldn't work. > > > 3 or 4 us now have mentioned curiosity in converting anaconda packages to > wheels (with specific interest in the non-python lib dependencies as > wheels). > Anyone who tries this, please post your success or lack thereof. I'm pretty > curious. I couldn't find a spec for the conda format files. If it's documented somewhere I'd be happy to try writing a converter. But it'd be useless for Python 3.3 on Windows because the conda binaries are built against the wrong version of the C runtime. Might be interesting on other platforms, though. Paul From dholth at gmail.com Tue Dec 3 22:46:49 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 3 Dec 2013 16:46:49 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Tue, Dec 3, 2013 at 4:13 PM, Donald Stufft wrote: > I think Wheels are the way forward for Python dependencies. Perhaps not for > things like fortran. I hope that the scientific community can start > publishing wheels at least in addition too. > > I don't believe that Conda will gain the mindshare that pip has outside of > the scientific community so I hope we don't end up with two systems that > can't interoperate. > > On Dec 2, 2013, at 7:00 PM, Chris Barker wrote: > > Side note about naming: > > I'm no expert, but I'm pretty sure "Anoconda" is a python distribution -- > python itself and set of pre-build packages. "conda" is the package manager > that is used by Anoconda -- kind of like rpm is used by RedHat -- conda is > an open-source project, and thus could be used by any of us completely apart > from the Anoconda distribution. > > > On Sun, Dec 1, 2013 at 3:38 PM, Paul Moore wrote: >> >> > had to resort to Google to try to figure out what dev libraries I >> > needed. >> >> But that's a *build* issue, surely? How does that relate to installing >> Nikola from a set of binary wheels? >> > Exactly -- I've mostly dealt with this for OS-X -- there are a cadre of > users that want binaries, and want them to "just work" -- we've had mpkg > packages for a good while, analogous to Windows installers. Binary eggs > never worked quite right, 'cause setuptools didn't understand "universal" > binaries -- but it wasn't that far from working. Not really tested much yet, > but it ;looks like binary wheels should be just fine. The concern there is > that someone will be running, say, a homebrew-built python, and accidentally > install a binary wheel built for the python.org python -- we should address > that with better platform tags (and making sure pip at least give a warning > if you try to install an incompatible wheel...) We are at least as worried about the homebrew user uploading a popular package as a binary wheel, and having it fail to work for the more common? non-homebrew user. > So what problem are we trying to solve here? > > 1) It's still a pain to actually build the packages -- similarly to Windows, > you really need to build the dependent libraries statically and link them in > - and you need to make sure that you build them with teh right sdk, and > universally -- this is hard to do right. > - does Conda help you do any of that??? > > 2) non-python binary dependencies: As it turns out, a number of python > packages depend on the same third-party non-python dependencies: I have > quite a few that use libpng, libfreetype, libhdf, ??? currently if you want > to distribute binary python packages, you need to statically link or supply > the dlls, so we end up with multiple coples of the same lib -- is this a > problem? Maybe not -- memory is pretty cheap these days, and maybe different > packages actually rely on different versions of the dependencies -- this > way, at least the package builder controls that. > > Anoconda (the distribution seems to address this by having conda packages > that are essentially containers for the shared libs, and other packages that > need those libs depend on them. I like this method, but it seems to me to be > more a feature of the Anoconda distribution than the conda package manager > -- in fact, I've been thinking of doing this exact same thing with binary > wheels -- I haven't tried it yet, but don't see why it wouldn't work. > >> I understand you are thinking about non-Python libraries, but all I >> can say is that this has *never* been an issue to my knowledge in the >> Windows world. > > > yes, it's a HUGE issue in the Windows world -- in fact such a huge issue > that almost non one ever tries to build things themselves, or build a > different python distro -- so, in fact, when someone does make a binary, > it's pretty likely to work. But those binaries are a major pain to build! > > (by the way, over on python-dev there has been a recent discussion about > stackless building a new python2.7 windows binary with a newer MS compiler > -- which will then create exacty these issues...) > >> > Outside the scientific space, crypto libraries are also notoriously hard >> > to >> > build, as are game engines and GUI toolkits. (I guess database bindings >> > could also be a problem in some cases) >> >> Build issues again... > > > Yes, major ones. > > (another side note: you can't get wxPython for OS-X to work with Anoconda -- > there is no conda binary package, and python itself is not built in a way > that it can access the window manager ... so no, this stuff in NOT suddenly > easier with conda.) > >> Again, can we please be clear here? On Windows, there is no issue that >> I am aware of. Wheels solve the binary distribution issue fine in that >> environment > > > They will if/when we make sure that the wheel contains meta-data about what > compiler (really run-time version) was used for the python build and wheel > build -- but we should, indeed, do that. > >> > This is why I suspect there will be a better near term effort/reward >> > trade-off in helping the conda folks improve the usability of their >> > platform >> > than there is in trying to expand the wheel format to cover arbitrary >> > binary >> > dependencies. > > > and have yet anoto=her way to do it? AARRG! I'm also absolutely unclear on > what conda offers that isn't quite easy to address with binary wheels. And > it seems to need help too, so it will play better with virtualenv.... > > If conda really is a better solution, then I suppose we could go deprecate > wheel before it gets too much "traction"...;-) But let's please not another > one to the mix to confuse people. > >> Excuse me if I'm feeling a bit negative towards this announcement. >> I've spent many months working on, and promoting, the wheel + pip >> solution, to the point where it is now part of Python 3.4. > > > And I was really lookingn forward to it as a solution for OS-X too.... > >> I'm hoping I've misunderstood here. Please clarify. Preferably with >> specifics for Windows (as "conda is a known stable platform" simply >> isn't true for me...) - I accept you're not a Windows user, so a >> pointer to already-existing documentation is fine (I couldn't find any >> myself). > > > I appreciate the Windows viewpoint -- and I think we really do want one > solution for all. > > The linux distros already do all this let them keep doing it.... > > -Chris > > PS: a number of scipy-related packages have been promoting Anoconda and > Canopy as a way to get their package without dealing with building (rather > than say, providing binary wheels). That works great for the SciPy Stack, > but has NOT worked well for others. My example: > > I'm teaching an Intro to Python class -- I really like iPython, so have been > using it for demos and recommend it to Students. The IPython web site makes > it look like you really need to go get Anaconda or Canopy if you want > iPython -- so a number of my students went and did that. All well. Until I > did the extra class on wxPython, and now I've got a bunch of students that > have no way to install it. And they mostly had no idea that they were > running a different Python at all... > > Anyway -- this is a mess, particularly on OS-X (now we have python binaries > from: Apple, python.org, fink, macports, homebrew, Anaconda, Canopy, ???) So > yes, we need a solution, but I think binary wheels are a pretty good one, > and I'm not sure what conda buys us... The most striking difference may be that conda also installs and manages Python itself. For example, conda create -n py33 python=3.3 will download and install Python 3.3 into a new environment named py33. This is completely different than pip which tends to run inside the same Python environment that it's installing into. Wheels were only designed to hold Python libraries to be installed into site-packages, with package names registered on pypi, and wheels were designed to allow the parts of the distribution to be relocated at install time as in putting scripts into Scripts\ or bin/ depending on the platform. If you wanted to interoperate you would probably install pip with "conda install pip" and then play with some wheels, or you could probably create a virtualenv from your conda environment and proceed. I would not recommend trying to package the CPython interpreter as a wheel. It wasn't designed for that. If you try let us know what happened :-) It's interesting that there is so much talk about compatibility with numpy-built-with-different-options. When I was thinking about the problem of binary compatibility I was more concerned about dynamically linking with different versions of system libraries (glibc) than the ones that the pre-built [wheel] package expects. In summary conda is very different than pip+virtualenv. From p.f.moore at gmail.com Tue Dec 3 22:47:27 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 3 Dec 2013 21:47:27 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 21:34, Marcus Smith wrote: >> The IPython web site makes it look like you really need to go get Anaconda >> or Canopy if you want iPython > > interesting... You really don't. I've been running IPython on Windows from wheels for ages. And you can actually build from source easily if you don't want the Qt console - the only component that includes C code is pyzmq and that builds out of the box if you have a C compiler. Paul. From donald at stufft.io Tue Dec 3 22:48:07 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Dec 2013 16:48:07 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: <7E248433-5F60-4128-BDAE-CAC9D2AF8641@stufft.io> On Dec 3, 2013, at 4:46 PM, Daniel Holth wrote: > In summary conda is very different than pip+virtualenv. Conda is a cross platform Homebrew. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Tue Dec 3 22:58:48 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Dec 2013 16:58:48 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <7E248433-5F60-4128-BDAE-CAC9D2AF8641@stufft.io> References: <7E248433-5F60-4128-BDAE-CAC9D2AF8641@stufft.io> Message-ID: <9A7445EC-085A-47CE-985B-EC3CC313C6F3@stufft.io> Filed https://github.com/ContinuumIO/conda-recipes/issues/42 :( On Dec 3, 2013, at 4:48 PM, Donald Stufft wrote: > > On Dec 3, 2013, at 4:46 PM, Daniel Holth wrote: > >> In summary conda is very different than pip+virtualenv. > > Conda is a cross platform Homebrew. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From qwcode at gmail.com Tue Dec 3 23:00:44 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 3 Dec 2013 14:00:44 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > > The most striking difference may be that conda also installs and > manages Python itself. For example, conda create -n py33 python=3.3 > will download and install Python 3.3 into a new environment named > py33. This is completely different than pip which tends to run inside > the same Python environment that it's installing into. > we've been talking about (and I've tried) "conda init" , not "conda create". that sure seems to setup conda in your *current* python. I had pip (the one that installed conda) and conda working in the same environment. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Dec 3 23:01:53 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 3 Dec 2013 14:01:53 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: "conda init" isn't in the website docs. On Tue, Dec 3, 2013 at 2:00 PM, Marcus Smith wrote: > > >> The most striking difference may be that conda also installs and >> manages Python itself. For example, conda create -n py33 python=3.3 >> will download and install Python 3.3 into a new environment named >> py33. This is completely different than pip which tends to run inside >> the same Python environment that it's installing into. >> > > we've been talking about (and I've tried) "conda init" , not "conda > create". > that sure seems to setup conda in your *current* python. > I had pip (the one that installed conda) and conda working in the same > environment. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Dec 3 23:18:15 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 3 Dec 2013 14:18:15 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Tue, Dec 3, 2013 at 12:48 AM, Nick Coghlan wrote: > Because it already works for the scientific stack, and if we don't provide > any explicit messaging around where conda fits into the distribution > picture, users are going to remain confused about it for a long time. > Do we have to have explicit messaging for every useful third-party package out there? > I'm still confused as to why packages need to share external dependencies > (though I can see why it's nice...) . > > Because they reference shared external data, communicate through shared > memory, or otherwise need compatible memory layouts. It's exactly the same > reason all C extensions need to be using the same C runtime as CPython on > Windows: because things like file descriptors break if they don't. > OK -- maybe we need a better term than shared external dependencies -- that makes me think shared library. Also even the scipy stack is not as dependent in build env as we seem to thin it is -- I don't think there is any reason you can't use the "standard" MPL with Golke's MKL-build numpy, for instance. And I"m pretty sure that even scipy and numpy don't need to share their build environment more than any other extension (i.e. they could use different BLAS implementations, etc... numpy version matters, but that's handled by the usual dependency handling. The reason Gohke's repo, and Anoconda and Canopy all exist is because it's a pain to build some of this stuff, period, not complex compatibly issues -- and the real pain goes beyond the standard scipy stack (VTK is a killer!) > Conda solves a specific problem for the scientific community, > well, we are getting Anaconda, the distribution, and conda, the package manager, conflated here: Having a nice full distribution of all the packages you are likely to need to great, but you could so that with wheels, and Gohlke is already doing it with MSIs (which don't handle dependencies at all -- whic is a problem). > but in their enthusiasm, the developers are pitching it as a general > purpose packaging solution. It isn't, > It's not? Aside from momentum, and all that, could it not be a replacement for pip and wheel? > Wheels *are* the way if one or both of the following conditions hold: > > - you don't need to deal with build variants > - you're building for a specific target environment > > That covers an awful lot of ground, but there's one thing it definitely > doesn't cover: distributing multiple versions of NumPy built with different > options and cohesive ecosystems on top of that. > hmm -- I'm not sure, you could have an Anoconda-like repo built with wheels, could you not? granted, it would be easier to make a mistake, and pull wheels from two different wheelhouses that are incompatible, so there is a real advantage to conda there. > By contrast, conda already exists, and already works, as it was designed > *specifically* to handle the scientific Python stack. > I'm not sure we how well it works -- it works for Anoconda, and good point about the scientifc stack -- does it work equally well for other stacks? or mixing and matching? > This means that one key reason I want to recommend it for the cases > where it is a good fit (i.e. the scientific Python stack) is so we can > explicitly advise *against* using it in other cases where it will just add > complexity without adding value. > I'm actually pretty concerned about this: lately the scipy community has defined a core "scipy stack": http://www.scipy.org/stackspec.html Along with this is a push to encourage users to just go with a scipy distribution to get that "stack": http://www.scipy.org/install.html and http://ipython.org/install.html I think this is in response to a years of pain of each package trying to build binaries for various platforms, and keeping it all in sync, etc. I feel their pain, and "just go with Anaconda or Canopy" is good advise for folks who want to get the "stack" up and running as easily as possible. But it does not server everyone else well -- web developers that need MPL for some plotting , scientific users that need a desktop GUI toolkit, pyhton newbies that want iPython, but none of that other stuff... What would serve all those folks well is a "standard build" of packages -- i.e. built to go with the python.org builds, that can be downloaded with: pip install the_package. And I think, with binary wheels, we have the tools to do that. > Saying nothing is not an option, since people are already confused. Saying > to never use it isn't an option either, since bootstrapping conda first > *is* a substantially simpler cross-platform way to get up to date > scientific Python software on to your system. > again, it is Anoconda that helps here, not conda itself. Or > how about a scientist that wants wxPython (to use Chris' example)? > Apparently the conda repo doesn't include wxPython, so do they need to > learn how to install pip into a conda environment? (Note that there's > no wxPython wheel, so this isn't a good example yet, but I'd hope it > will be in due course...) Actually if only it were as simple as "install pip', but as you point out, there is no wxPython binary wheel, but if there were, it would be compatible with the pyton.org python, and maybe not Anoconda (would conda catch that?) Looks like the conda stack is built around msvcr90, whereas python.org > Python 3.3 is built around msvcr100. > So conda is not interoperable *at all* with standard python.org Python > 3.3 on Windows :-( again, Anaconda the distribution, is not, but I assume conda, the package manager, is. And IIUC, then conda would catch that incompatibly if you tried to install incompatible packages. That's the whole point, yes? And this would help the recent concerns from the stackless folks about building a pyton binary for Windows with a newer MSVC (see pyton-dev) - if there's no wheel > - and you can't build it from source yourself > - then you can try "pip install conda && conda init && conda install > " as a fallback option. > And then we encourage the conda devs to follow the installation > database standard properly (if they aren't already), so things > installed with conda play nice with things installed with pip. > It sounds like we also need to get them to ensure they're using the > right compiler/C runtime on Windows so their packages are > interoperable with the standard python.org installers. maybe we should just have conda talk to PyPi? As it stands, one of the POINTS of Anoconda is that it ISN'T the standard pyhton.org installer! But really, this just puts us back in the state that we want to avoid -- a bunch of binary-incompatible builds out there to get confused by -- again thouhg, at least conda apparently wont let you install binary incompatible packages... However, there haven't been any compelling examples presented other > than the C runtime (which wheel needs to handle as part of the > platform tag and/or the ABI tag) and the scientific stack, Again, I'm pretty sure it doesn't even apply to the scientific stack in any special way... At the moment, we're getting people trying to use conda as the base, > and stuff falling apart at a later stage, Still confused: conda the package manager, or Anaconda the distribution? well, except that the anaconda index covers non-python projects like "qt", > which a private wheel index wouldn't cover (at least with the normal > intended use of wheels) umm, why not? you couldn't have a pySide wheel??? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Dec 4 00:50:03 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 3 Dec 2013 15:50:03 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: > > well, except that the anaconda index covers non-python projects like "qt", >> which a private wheel index wouldn't cover (at least with the normal >> intended use of wheels) > > > umm, why not? you couldn't have a pySide wheel??? > just saying that the anaconda index literally has packages for "qt" itself, the c++ library. http://repo.continuum.io/pkgs/free/linux-64/qt-4.8.5-0.tar.bz2 and it's pyside packages require that. my understanding is that you could build a pyside wheel that was statically linked to qt. as to whether a wheel could just package "qt". that's what I don't know, and if it could, the wheel spec doesn't cover that use case. > > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Dec 4 01:02:57 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 3 Dec 2013 16:02:57 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Tue, Dec 3, 2013 at 3:50 PM, Marcus Smith wrote: > >> umm, why not? you couldn't have a pySide wheel??? >> > > just saying that the anaconda index literally has packages for "qt" > itself, the c++ library. > http://repo.continuum.io/pkgs/free/linux-64/qt-4.8.5-0.tar.bz2 > > and it's pyside packages require that. > > That appears to be how Anaconda does things -- creates a conda package that hols just the shared libs, then another one that holds the python packages that use those libs. (I think there is a freetype one, for instance) It's a nice way to let multiple packages share the same dynamic libs, while still keeping them part of the packaging and dependency system. But I don't see why you couldn't do the same thing with wheels... (and I'm not sure there's a point if there is nothing else that uses the same libs...) > my understanding is that you could build a pyside wheel that was > statically linked to qt. > > which is how it's usually done for stuff like that -- or a lot of shared libs are bundles in. > as to whether a wheel could just package "qt". that's what I don't know, > and if it could, the wheel spec doesn't cover that use case. > you'd have to make up a "fake" python package -- but it wouldn't have to have anything in it (Or not much...) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed Dec 4 01:10:53 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Dec 2013 00:10:53 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 22:18, Chris Barker wrote: > On Tue, Dec 3, 2013 at 12:48 AM, Nick Coghlan wrote: >> >> Because it already works for the scientific stack, and if we don't provide >> any explicit messaging around where conda fits into the distribution >> picture, users are going to remain confused about it for a long time. > > Do we have to have explicit messaging for every useful third-party package > out there? > >> > I'm still confused as to why packages need to share external >> > dependencies (though I can see why it's nice...) . >> >> Because they reference shared external data, communicate through shared >> memory, or otherwise need compatible memory layouts. It's exactly the same >> reason all C extensions need to be using the same C runtime as CPython on >> Windows: because things like file descriptors break if they don't. > > OK -- maybe we need a better term than shared external dependencies -- that > makes me think shared library. Also even the scipy stack is not as dependent > in build env as we seem to thin it is -- I don't think there is any reason > you can't use the "standard" MPL with Golke's MKL-build numpy, for instance. > And I"m pretty sure that even scipy and numpy don't need to share their > build environment more than any other extension (i.e. they could use > different BLAS implementations, etc... numpy version matters, but that's > handled by the usual dependency handling. Sorry, I was being vague earlier. The BLAS information is not important but the Fortran ABI it exposes is: http://docs.scipy.org/doc/numpy/user/install.html#fortran-abi-mismatch MPL - matplotlib for those unfamiliar with the acronym - depends on the numpy C API/ABI but not the Fortran ABI. So it would be incompatible with, say, a pure Python implementation of numpy (or with numpypy) but it should work fine with any of the numpy binaries currently out there. (Numpy's C ABI has been unchanged from version 1.0 to 1.7 precisely because changing it has been too too painful to contemplate). > The reason Gohke's repo, and Anoconda and Canopy all exist is because it's a > pain to build some of this stuff, period, not complex compatibly issues -- > and the real pain goes beyond the standard scipy stack (VTK is a killer!) I agree that the binary compatibility issues are not as complex as some are making out but it is a fact that his binaries are sometimes binary-incompatible with other builds. I have seen examples of it going wrong and he gives a clear warning at the top of his downloads page: http://www.lfd.uci.edu/~gohlke/pythonlibs/ >> but in their enthusiasm, the developers are pitching it as a general >> purpose packaging solution. It isn't, > > It's not? Aside from momentum, and all that, could it not be a replacement > for pip and wheel? Conda/binstar could indeed be a replacement for pip and wheel and PyPI. It currently lacks many packages but less so than PyPI if you're mainly interested in binaries. For me pip+PyPI is a non-starter (as a complete solution) if I can't install numpy and matplotlib. >> By contrast, conda already exists, and already works, as it was designed >> *specifically* to handle the scientific Python stack. > > I'm not sure we how well it works -- it works for Anoconda, and good point > about the scientifc stack -- does it work equally well for other stacks? or > mixing and matching? I don't even know how well it works for the "scientific stack". It didn't work for me! But I definitely know that pip+PyPI doesn't yet work for me and working around that has caused me a lot more pain then it would be to diagnose and fix the problem I had with conda. They might even accept a one line, no-brainer pull request for my fix in less then 3 months :) https://github.com/pypa/pip/pull/1187 >> This means that one key reason I want to recommend it for the cases where >> it is a good fit (i.e. the scientific Python stack) is so we can explicitly >> advise *against* using it in other cases where it will just add complexity >> without adding value. > > I'm actually pretty concerned about this: lately the scipy community has > defined a core "scipy stack": > > http://www.scipy.org/stackspec.html > > Along with this is a push to encourage users to just go with a scipy > distribution to get that "stack": > > http://www.scipy.org/install.html > > and > > http://ipython.org/install.html > > I think this is in response to a years of pain of each package trying to > build binaries for various platforms, and keeping it all in sync, etc. I > feel their pain, and "just go with Anaconda or Canopy" is good advise for > folks who want to get the "stack" up and running as easily as possible. The scientific Python community are rightfully worried about potential users losing interest in Python because these installation problems occur for every noob who wants to use Python. In scientific usage Python just isn't fully installed yet until numpy/scipy/matplotlib etc. is. It makes perfect sense to try and get people introduced to Python for scientific use in a way that minimises (delays?) their encounter with the packaging problems in the Python ecosystem. > But it does not server everyone else well -- web developers that need MPL > for some plotting , scientific users that need a desktop GUI toolkit, pyhton > newbies that want iPython, but none of that other stuff... > > What would serve all those folks well is a "standard build" of packages -- > i.e. built to go with the python.org builds, that can be downloaded with: > > pip install the_package. > > And I think, with binary wheels, we have the tools to do that. Yes but there will never (rather not any time soon) be a single universally compatible binary configuration even for say Windows 32 bit. I think that the binary wheels do need more compatibility information. You mentioned C runtimes, also Fortran ABIs. But I don't think the solution is for a PEP to enumerate the possibilities - although it might be worth having an official stance on C runtimes for Windows and for POSIX. I think the solution is to have community-extensible compatibility information. [snip] > > maybe we should just have conda talk to PyPi? > > As it stands, one of the POINTS of Anoconda is that it ISN'T the standard > pyhton.org installer! Actually I think conda does (or will soon) just invoke pip under the hood for packages that aren't in the binstar/Anaconda channels but are on PyPI. Oscar From oscar.j.benjamin at gmail.com Wed Dec 4 01:36:20 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Dec 2013 00:36:20 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 21:13, Donald Stufft wrote: > I think Wheels are the way forward for Python dependencies. Perhaps not for > things like fortran. I hope that the scientific community can start > publishing wheels at least in addition too. The Fortran issue is not that complicated. Very few packages are affected by it. It can easily be fixed with some kind of compatibility tag that can be used by the small number of affected packages. > I don't believe that Conda will gain the mindshare that pip has outside of > the scientific community so I hope we don't end up with two systems that > can't interoperate. Maybe conda won't gain mindshare outside the scientific community but wheel really needs to gain mindshare *within* the scientific community. The root of all this is numpy. It is the biggest dependency on PyPI, is hard to build well, and has the Fortran ABI issue. It is used by very many people who wouldn't consider themselves part of the "scientific community". For example matplotlib depends on it. The PyPy devs have decided that it's so crucial to the success of PyPy that numpy's basically being rewritten in their stdlib (along with the C API). A few times I've seen Paul Moore refer to numpy as the "litmus test" for wheels. I actually think that it's more important than that. If wheels are going to fly then there *needs* to be wheels for numpy. As long as there isn't a wheel for numpy then there will be lots of people looking for a non-pip/PyPI solution to their needs. One way of getting the scientific community more on board here would be to offer them some tangible advantages. So rather than saying "oh well scientific use is a special case so they should just use conda or something", the message should be "the wheel system provides solutions to many long-standing problems and is even better than conda in (at least) some ways because it cleanly solves the Fortran ABI issue for example". Oscar From donald at stufft.io Wed Dec 4 01:54:51 2013 From: donald at stufft.io (Donald Stufft) Date: Tue, 3 Dec 2013 19:54:51 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Dec 3, 2013, at 7:36 PM, Oscar Benjamin wrote: > On 3 December 2013 21:13, Donald Stufft wrote: >> I think Wheels are the way forward for Python dependencies. Perhaps not for >> things like fortran. I hope that the scientific community can start >> publishing wheels at least in addition too. > > The Fortran issue is not that complicated. Very few packages are > affected by it. It can easily be fixed with some kind of compatibility > tag that can be used by the small number of affected packages. > >> I don't believe that Conda will gain the mindshare that pip has outside of >> the scientific community so I hope we don't end up with two systems that >> can't interoperate. > > Maybe conda won't gain mindshare outside the scientific community but > wheel really needs to gain mindshare *within* the scientific > community. The root of all this is numpy. It is the biggest dependency > on PyPI, is hard to build well, and has the Fortran ABI issue. It is > used by very many people who wouldn't consider themselves part of the > "scientific community". For example matplotlib depends on it. The PyPy > devs have decided that it's so crucial to the success of PyPy that > numpy's basically being rewritten in their stdlib (along with the C > API). > > A few times I've seen Paul Moore refer to numpy as the "litmus test" > for wheels. I actually think that it's more important than that. If > wheels are going to fly then there *needs* to be wheels for numpy. As > long as there isn't a wheel for numpy then there will be lots of > people looking for a non-pip/PyPI solution to their needs. > > One way of getting the scientific community more on board here would > be to offer them some tangible advantages. So rather than saying "oh > well scientific use is a special case so they should just use conda or > something", the message should be "the wheel system provides solutions > to many long-standing problems and is even better than conda in (at > least) some ways because it cleanly solves the Fortran ABI issue for > example". > > > Oscar I?d love to get Wheels to the point they are more suitable then they are for SciPy stuff, I?m not sure what the diff between the current state and what they need to be are but if someone spells it out (I?ve only just skimmed your last email so perhaps it?s contained in that!) I?ll do the arguing for it. I just need someone who actually knows what?s needed to advise me :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From ralf.gommers at gmail.com Wed Dec 4 08:40:22 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 4 Dec 2013 08:40:22 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: > > On Dec 3, 2013, at 7:36 PM, Oscar Benjamin > wrote: > > > On 3 December 2013 21:13, Donald Stufft wrote: > >> I think Wheels are the way forward for Python dependencies. Perhaps not > for > >> things like fortran. I hope that the scientific community can start > >> publishing wheels at least in addition too. > > > > The Fortran issue is not that complicated. Very few packages are > > affected by it. It can easily be fixed with some kind of compatibility > > tag that can be used by the small number of affected packages. > > > >> I don't believe that Conda will gain the mindshare that pip has outside > of > >> the scientific community so I hope we don't end up with two systems that > >> can't interoperate. > > > > Maybe conda won't gain mindshare outside the scientific community but > > wheel really needs to gain mindshare *within* the scientific > > community. The root of all this is numpy. It is the biggest dependency > > on PyPI, is hard to build well, and has the Fortran ABI issue. It is > > used by very many people who wouldn't consider themselves part of the > > "scientific community". For example matplotlib depends on it. The PyPy > > devs have decided that it's so crucial to the success of PyPy that > > numpy's basically being rewritten in their stdlib (along with the C > > API). > > > > A few times I've seen Paul Moore refer to numpy as the "litmus test" > > for wheels. I actually think that it's more important than that. If > > wheels are going to fly then there *needs* to be wheels for numpy. As > > long as there isn't a wheel for numpy then there will be lots of > > people looking for a non-pip/PyPI solution to their needs. > > > > One way of getting the scientific community more on board here would > > be to offer them some tangible advantages. So rather than saying "oh > > well scientific use is a special case so they should just use conda or > > something", the message should be "the wheel system provides solutions > > to many long-standing problems and is even better than conda in (at > > least) some ways because it cleanly solves the Fortran ABI issue for > > example". > > > > > > Oscar > > I?d love to get Wheels to the point they are more suitable then they are > for > SciPy stuff, That would indeed be a good step forward. I'm interested to try to help get to that point for Numpy and Scipy. I?m not sure what the diff between the current state and what > they need to be are but if someone spells it out (I?ve only just skimmed > your last email so perhaps it?s contained in that!) I?ll do the arguing > for it. I > just need someone who actually knows what?s needed to advise me :) > To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" installers for Windows containing three full builds: no SSE, SSE2 and SSE3. Plus a script that runs at install time to check which version to use. These are built with ``paver bdist_superpack``, see https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and CPU selector scripts are under tools/win32build/. How do I package those three builds into wheels and get the right one installed by ``pip install numpy``? If this is too difficult at the moment, an easier (but much less important one) would be to get the result of ``paver bdist_wininst_simple`` as a wheel. For now I think it's OK that the wheels would just target 32-bit Windows and python.org compatible Pythons (given that that's all we currently distribute). Once that works we can look at OS X and 64-bit Windows. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Dec 4 09:04:22 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Dec 2013 08:04:22 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 3 December 2013 22:18, Chris Barker wrote: >> Looks like the conda stack is built around msvcr90, whereas python.org >> Python 3.3 is built around msvcr100. >> So conda is not interoperable *at all* with standard python.org Python >> 3.3 on Windows :-( > > again, Anaconda the distribution, is not, but I assume conda, the package > manager, is. And IIUC, then conda would catch that incompatibly if you tried > to install incompatible packages. That's the whole point, yes? And this > would help the recent concerns from the stackless folks about building a > pyton binary for Windows with a newer MSVC (see pyton-dev) conda the installer only looks in the Anaconda repos (at the moment, and by default - you can add your own conda-format repos if you have any). So no, this *is* a problem with conda, not just Anaconda. And no, it doesn't catch the incompatibility, which says something about the robustness of their compatibility checking solution, I guess... Paul From p.f.moore at gmail.com Wed Dec 4 09:13:08 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Dec 2013 08:13:08 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 07:40, Ralf Gommers wrote: >> I?m not sure what the diff between the current state and what >> they need to be are but if someone spells it out (I?ve only just skimmed >> your last email so perhaps it?s contained in that!) I?ll do the arguing >> for it. I >> just need someone who actually knows what?s needed to advise me :) > > > To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" > installers for Windows containing three full builds: no SSE, SSE2 and SSE3. > Plus a script that runs at install time to check which version to use. These > are built with ``paver bdist_superpack``, see > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and > CPU selector scripts are under tools/win32build/. > > How do I package those three builds into wheels and get the right one > installed by ``pip install numpy``? I think that needs a compatibility tag. Certainly it isn't immediately soluble now. Could you confirm how the correct one of the 3 builds is selected (i.e., what the code is to detect which one is appropriate)? I could look into what options we have here. > If this is too difficult at the moment, an easier (but much less important > one) would be to get the result of ``paver bdist_wininst_simple`` as a > wheel. That I will certainly look into. Simple answer is "wheel convert ". But maybe it would be worth adding a "paver bdist_wheel" command. That should be doable in the same wahy setuptools added a bdist_wheel command. > For now I think it's OK that the wheels would just target 32-bit Windows and > python.org compatible Pythons (given that that's all we currently > distribute). Once that works we can look at OS X and 64-bit Windows. Ignoring the SSE issue, I believe that simply wheel converting Christoph Gohlke's repository gives you that right now. The only issues there are (1) the MKL license limitation, (2) hosting, and (3) whether Christoph would be OK with doing this (he goes to lengths on his site to prevent spidering his installers). I genuinely believe that "a schientific stack for non-scientists" is trivially solved in this way. For scientists, of course, we'd need to look deeper, but having a base to start from would be great. Paul From p.f.moore at gmail.com Wed Dec 4 10:33:31 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Dec 2013 09:33:31 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 08:13, Paul Moore wrote: >> If this is too difficult at the moment, an easier (but much less important >> one) would be to get the result of ``paver bdist_wininst_simple`` as a >> wheel. > > That I will certainly look into. Simple answer is "wheel convert > ". But maybe it would be worth adding a "paver bdist_wheel" > command. That should be doable in the same wahy setuptools added a > bdist_wheel command. Actually, I just installed paver and wheel into a virtualenv, converted a trivial project to use paver, and ran "paver bdist_wheel" and it worked out of the box. I don't know if there could be problems with more complex projects, but if you hit any issues, flag them up and I'll take a look. Paul From oscar.j.benjamin at gmail.com Wed Dec 4 11:41:41 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Dec 2013 10:41:41 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 07:40, Ralf Gommers wrote: > On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: >> >> I?d love to get Wheels to the point they are more suitable then they are >> for SciPy stuff, > > That would indeed be a good step forward. I'm interested to try to help get > to that point for Numpy and Scipy. Thanks Ralf. Please let me know what you think of the following. >> I?m not sure what the diff between the current state and what >> they need to be are but if someone spells it out (I?ve only just skimmed >> your last email so perhaps it?s contained in that!) I?ll do the arguing >> for it. I >> just need someone who actually knows what?s needed to advise me :) > > To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" > installers for Windows containing three full builds: no SSE, SSE2 and SSE3. > Plus a script that runs at install time to check which version to use. These > are built with ``paver bdist_superpack``, see > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and > CPU selector scripts are under tools/win32build/. > > How do I package those three builds into wheels and get the right one > installed by ``pip install numpy``? This was discussed previously on this list: https://mail.python.org/pipermail/distutils-sig/2013-August/022362.html Essentially the current wheel format and specification does not provide a way to do this directly. There are several different possible approaches. One possibility is that the wheel spec can be updated to include a post-install script (I believe this will happen eventually - someone correct me if I'm wrong). Then the numpy for Windows wheel can just do the same as the superpack installer: ship all variants, then delete/rename in a post-install script so that the correct variant is in place after install. Another possibility is that the pip/wheel/PyPI/metadata system can be changed to allow a "variant" field for wheels/sdists. This was also suggested in the same thread by Nick Coghlan: https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html The variant field could be used to upload multiple variants e.g. numpy-1.7.1-cp27-cp22m-win32.whl numpy-1.7.1-cp27-cp22m-win32-sse.whl numpy-1.7.1-cp27-cp22m-win32-sse2.whl numpy-1.7.1-cp27-cp22m-win32-sse3.whl then if the user requests 'numpy:sse3' they will get the wheel with sse3 support. Of course how would the user know if their CPU supports SSE3? I know roughly what SSE is but I don't know what level of SSE is avilable on each of the machines I use. There is a Python script/module in numpexpr that can detect this: https://github.com/eleddy/numexpr/blob/master/numexpr/cpuinfo.py When I run that script on this machine I get: $ python cpuinfo.py CPU information: CPUInfoBase__get_nbits=32 getNCPUs=2 has_mmx has_sse2 is_32bit is_Core2 is_Intel is_i686 So perhaps someone could break that script out of numexpr and release it as a separate package on PyPI. Then the instructions for installing numpy could be something like """ You can install numpy with $pip install numpy which will download the default version without any CPU-specific optimisations. If you know what level of SSE support your CPU has then you can download a more optimised numpy with either of: $ pip install numpy:sse2 $ pip install numpy:sse3 To determine whether or not your CPU has SSE2 or SSE3 or no SSE support you can install and run the cpuinfo script. For example on this machine: $ pip install cpuinfo $ python -m cpuinfo --sse This CPU supports the SSE3 instruction set. That means we can install numpy:sse3. """ Of course it would be a shame to have a solution that is so close to automatic without quite being automatic. Also the problem is that having no SSE support in the default numpy means that lots of people would lose out on optimisations. For example if numpy is installed as a dependency of something else then the user would always end up with the unoptimised no-SSE binary. Another possibility is that numpy could depend on the cpuinfo package so that it gets installed automatically before numpy. Then if the cpuinfo package has a traditional setup.py sdist (not a wheel) it could detect the CPU information at install time and store that in its package metadata. Then pip would be aware of this metadata and could use it to determine which wheel is appropriate. I don't quite know if this would work but perhaps the cpuinfo could announce that it "Provides" e.g. cpuinfo:sse2. Then a numpy wheel could "Requires" cpuinfo:sse2 or something along these lines. Or perhaps this is better handled by the metadata extensions Nick suggested earlier in this thread. I think it would be good to work out a way of doing this with e.g. a cpuinfo package. Many other packages beyond numpy could make good use of that metadata if it were available. Similarly having an extensible mechanism for selecting wheels based on additional information about the user's system could be used for many more things than just CPU architectures. Oscar From ncoghlan at gmail.com Wed Dec 4 12:10:06 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Dec 2013 21:10:06 +1000 Subject: [Distutils] Binary dependency management, round 2 :) Message-ID: There was some really good feedback in the binary dependency thread, but it ended up going off on a few different tangents. Rather than expecting people to read the whole thing, I figured I'd try to come up with a summary of where it has gone so far, and where we might want to take it from here. Given the diversity of topics, this should arguably be multiple new threads, but that approach has its own problems, since some of these topics are at least somewhat interrelated. In several cases, some of the discussion could possible move to the tracker issues I created :) == Regarding documentation == One of the things I got from the thread is that we don't currently have a clear overview published anywhere of *why* people use binary extensions. The distutils docs and both dated and focus solely on the mechanics, without discussing either use cases, or the benefits and limitations of using them. We also don't have a good location to describe the approach of statically linking or bundling additional libraries into "platlib" to avoid issues with external dependencies and incompatible ABI changes when distributing wheel files. This seems like something that would be suitably covered as an "Advanced Topic" in the packaging user's guide, so I filed an issue with some more specific ideas: https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-section-that-covers-binary == Regarding conda == In terms of providing an answer to the question "Where does conda fit in the scheme of packaging tools?", my conclusion from the thread is that once a couple of security related issues are fixed (think PyPI before the rubygems.org compromise for the current state of conda's security model), and once the Python 3.3 compatibility issue is addressed on Windows, it would be reasonable to recommend it as one of the current options for getting hold of pre-built versions of the scientific Python stack. I think this is important enough to warrant a "NumPy and the Scientific Python stack" section in the user guide (with Linux distro packages, Windows installers and conda all discussed as options): https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific == Regarding alternative index servers == Something I believe conda supports is the idea of installer configuration settings that are specific to a particular virtual environment (I can't find specific docs confirming that capability, but I certainly got that impression somewhere along the line). At the moment, it isn't straightforward to tell pip "when in this virtual environment, use these additional settings", but I believe such a feature could prove useful in dealing with some of the thornier binary compatibility problems. In particular, it would be good to be able to lock an environment to a particular custom index server that it will use instead of defaulting to PyPI. I've posted this idea to the pip issue tracker: https://github.com/pypa/pip/issues/1362 == Regarding NumPy, build variants and API/ABI consistency == My current reading of the NumPy situation is that it boils down to needing two additional features: - a richer platform tagging mechanism (which we need for *nix systems anyway) - a way to ensure internal consistency of the installed *builds* in an environment, not just the abstract dependencies I've opened a wheel format definition issue for the first problem: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for I've posted a metadata 2.0 standard extension proposal for the latter: https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-based-consistency-checking (the alternative index server idea is also relevant, since PyPI wouldn't support hosting multiple variants with the same wheel level compatibility tags) == Regarding custom installation directories == This technically came up in the cobblerd thread (regarding installing scripts to /usr/sbin instead of /usr/bin), but I believe it may also be relevant to the problem of shipping external libraries inside wheels, static data files for applications, etc. It's a little underspecified in PEP 427, but the way the wheel format currently handles installation to paths other than purelib and platlib (or to install to both of those as part of the same wheel) is to use the sysconfig scheme names as subdirectories within the wheel's .data directory. This approach is great for making it easy to build well-behaved cross platform wheels that play nice with virtual environments, but allowing a "just put it here" escape clause could potentially be a useful approach for platform specific wheels (especially on *nix systems that use the Filesystem Hierarchy Standard). I've posted this idea to the metadata format issue tracker: https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Dec 4 13:10:15 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Dec 2013 22:10:15 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 20:41, Oscar Benjamin wrote: > On 4 December 2013 07:40, Ralf Gommers wrote: >> On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: >>> >>> I?d love to get Wheels to the point they are more suitable then they are >>> for SciPy stuff, >> >> That would indeed be a good step forward. I'm interested to try to help get >> to that point for Numpy and Scipy. > > Thanks Ralf. Please let me know what you think of the following. > >>> I?m not sure what the diff between the current state and what >>> they need to be are but if someone spells it out (I?ve only just skimmed >>> your last email so perhaps it?s contained in that!) I?ll do the arguing >>> for it. I >>> just need someone who actually knows what?s needed to advise me :) >> >> To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" >> installers for Windows containing three full builds: no SSE, SSE2 and SSE3. >> Plus a script that runs at install time to check which version to use. These >> are built with ``paver bdist_superpack``, see >> https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and >> CPU selector scripts are under tools/win32build/. >> >> How do I package those three builds into wheels and get the right one >> installed by ``pip install numpy``? > > This was discussed previously on this list: > https://mail.python.org/pipermail/distutils-sig/2013-August/022362.html > > Essentially the current wheel format and specification does not > provide a way to do this directly. There are several different > possible approaches. > > One possibility is that the wheel spec can be updated to include a > post-install script (I believe this will happen eventually - someone > correct me if I'm wrong). Then the numpy for Windows wheel can just do > the same as the superpack installer: ship all variants, then > delete/rename in a post-install script so that the correct variant is > in place after install. Yes, export hooks in metadata 2.0 would support this approach. However, export hooks require allowing just-downloaded code to run with elevated privileges, so we're trying to minimise the number of cases where they're needed. > Another possibility is that the pip/wheel/PyPI/metadata system can be > changed to allow a "variant" field for wheels/sdists. This was also > suggested in the same thread by Nick Coghlan: > https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html > > The variant field could be used to upload multiple variants e.g. > numpy-1.7.1-cp27-cp22m-win32.whl > numpy-1.7.1-cp27-cp22m-win32-sse.whl > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > numpy-1.7.1-cp27-cp22m-win32-sse3.whl > then if the user requests 'numpy:sse3' they will get the wheel with > sse3 support. That was what I was originally thinking for the variant field, but I later realised it makes more sense to treat the "variant" marker as part of the *platform* tag, rather than being an independent tag in its own right: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for Under that approach, pip would figure out all the variants that applied to the current system (with some default preference order between variants for platforms where one system may support multiple variants). Using the Linux distro variants (based on ID and RELEASE_ID in /etc/os-release) as an example rather than the Windows SSE variants, this might look like: cp33-cp33m-linux_x86_64_fedora_19 cp33-cp33m-linux_x86_64_fedora cp33-cp33m-linux_x86_64 The Windows SSE variants might look like: cp33-cp33m-win32_sse3 cp33-cp33m-win32_sse2 cp33-cp33m-win32_sse cp33-cp33m-win32 > Of course how would the user know if their CPU supports SSE3? I know > roughly what SSE is but I don't know what level of SSE is avilable on > each of the machines I use. Asking this question is how I realised the variant tag should probably be part of the platform field and handled automatically by pip rather than users needing to request it explicitly. However, it's not without its problems (more on that below) > There is a Python script/module in > numpexpr that can detect this: > https://github.com/eleddy/numexpr/blob/master/numexpr/cpuinfo.py > > When I run that script on this machine I get: > $ python cpuinfo.py > CPU information: CPUInfoBase__get_nbits=32 getNCPUs=2 has_mmx has_sse2 > is_32bit is_Core2 is_Intel is_i686 > > So perhaps someone could break that script out of numexpr and release > it as a separate package on PyPI. Then the instructions for installing > numpy could be something like > """ > You can install numpy with > > $pip install numpy > > which will download the default version without any CPU-specific optimisations. > > If you know what level of SSE support your CPU has then you can > download a more optimised numpy with either of: > > $ pip install numpy:sse2 > $ pip install numpy:sse3 > > To determine whether or not your CPU has SSE2 or SSE3 or no SSE > support you can install and run the cpuinfo script. For example on > this machine: > > $ pip install cpuinfo > $ python -m cpuinfo --sse > This CPU supports the SSE3 instruction set. > > That means we can install numpy:sse3. > """ > > Of course it would be a shame to have a solution that is so close to > automatic without quite being automatic. Also the problem is that > having no SSE support in the default numpy means that lots of people > would lose out on optimisations. For example if numpy is installed as > a dependency of something else then the user would always end up with > the unoptimised no-SSE binary. The other question I asked that made me realise the SSE information should be an optional part of the platform tag :) > Another possibility is that numpy could depend on the cpuinfo package > so that it gets installed automatically before numpy. Then if the > cpuinfo package has a traditional setup.py sdist (not a wheel) it > could detect the CPU information at install time and store that in its > package metadata. Then pip would be aware of this metadata and could > use it to determine which wheel is appropriate. > > I don't quite know if this would work but perhaps the cpuinfo could > announce that it "Provides" e.g. cpuinfo:sse2. Then a numpy wheel > could "Requires" cpuinfo:sse2 or something along these lines. Or > perhaps this is better handled by the metadata extensions Nick > suggested earlier in this thread. > > I think it would be good to work out a way of doing this with e.g. a > cpuinfo package. Many other packages beyond numpy could make good use > of that metadata if it were available. Similarly having an extensible > mechanism for selecting wheels based on additional information about > the user's system could be used for many more things than just CPU > architectures. Yes, the lack of extensibility is the one concern I have with baking the CPU SSE info into the platform tag. On the other hand, the CPU architecture info is already in there, so appending the vectorisation support isn't an obviously bad idea, is orthogonal to the "python.expects" consistency enforcement metadata and would cover the NumPy use case, which is the one we really care about at this point. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From theller at ctypes.org Wed Dec 4 13:13:04 2013 From: theller at ctypes.org (Thomas Heller) Date: Wed, 04 Dec 2013 13:13:04 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: Am 04.12.2013 11:41, schrieb Oscar Benjamin: > On 4 December 2013 07:40, Ralf Gommers wrote: >> How do I package those three builds into wheels and get the right one >> installed by ``pip install numpy``? > > This was discussed previously on this list: > https://mail.python.org/pipermail/distutils-sig/2013-August/022362.html > > Essentially the current wheel format and specification does not > provide a way to do this directly. There are several different > possible approaches. > > One possibility is that the wheel spec can be updated to include a > post-install script (I believe this will happen eventually - someone > correct me if I'm wrong). Then the numpy for Windows wheel can just do > the same as the superpack installer: ship all variants, then > delete/rename in a post-install script so that the correct variant is > in place after install. > > Another possibility is that the pip/wheel/PyPI/metadata system can be > changed to allow a "variant" field for wheels/sdists. This was also > suggested in the same thread by Nick Coghlan: > https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html > > The variant field could be used to upload multiple variants e.g. > numpy-1.7.1-cp27-cp22m-win32.whl > numpy-1.7.1-cp27-cp22m-win32-sse.whl > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > numpy-1.7.1-cp27-cp22m-win32-sse3.whl > then if the user requests 'numpy:sse3' they will get the wheel with > sse3 support. Why does numpy not create a universal distribution, where the actual extensions used are determined at runtime? This would simplify the installation (all the stuff that you describe would not be required). Another benefit would be for users that create and distribute 'frozen' executables (py2exe, py2app, cx_freeze, pyinstaller), the exe would work on any machine independend from the sse - level. Thomas From oscar.j.benjamin at gmail.com Wed Dec 4 13:50:53 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 4 Dec 2013 12:50:53 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 12:10, Nick Coghlan wrote: > On 4 December 2013 20:41, Oscar Benjamin wrote: >> >> Another possibility is that the pip/wheel/PyPI/metadata system can be >> changed to allow a "variant" field for wheels/sdists. This was also >> suggested in the same thread by Nick Coghlan: >> https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html >> >> The variant field could be used to upload multiple variants e.g. >> numpy-1.7.1-cp27-cp22m-win32.whl >> numpy-1.7.1-cp27-cp22m-win32-sse.whl >> numpy-1.7.1-cp27-cp22m-win32-sse2.whl >> numpy-1.7.1-cp27-cp22m-win32-sse3.whl >> then if the user requests 'numpy:sse3' they will get the wheel with >> sse3 support. > > That was what I was originally thinking for the variant field, but I > later realised it makes more sense to treat the "variant" marker as > part of the *platform* tag, rather than being an independent tag in > its own right: https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for > > Under that approach, pip would figure out all the variants that > applied to the current system (with some default preference order > between variants for platforms where one system may support multiple > variants). Using the Linux distro variants (based on ID and RELEASE_ID > in /etc/os-release) as an example rather than the Windows SSE > variants, this might look like: > > cp33-cp33m-linux_x86_64_fedora_19 > cp33-cp33m-linux_x86_64_fedora > cp33-cp33m-linux_x86_64 I find that a bit strange to look at since I expect it to be like a taxonomic hierarchy like so: cp33-cp33m-linux cp33-cp33m-linux_fedora cp33-cp33m-linux_fedora_19 cp33-cp33m-linux_fedora_19_x86_64 Really you always need the architecture information though so cp33-cp33m-linux_x86_64 cp33-cp33m-linux_fedora_x86_64 cp33-cp33m-linux_fedora_19_x86_64 > The Windows SSE variants might look like: > > cp33-cp33m-win32_sse3 > cp33-cp33m-win32_sse2 > cp33-cp33m-win32_sse > cp33-cp33m-win32 I would have thought something like: cp33-cp33m-win32 cp33-cp33m-win32_nt cp33-cp33m-win32_nt_vista cp33-cp33m-win32_nt_vista_sp2 Also CPU information isn't hierarchical, so what happens when e.g. pyfftw wants to ship wheels with and without MMX instructions? >> I think it would be good to work out a way of doing this with e.g. a >> cpuinfo package. Many other packages beyond numpy could make good use >> of that metadata if it were available. Similarly having an extensible >> mechanism for selecting wheels based on additional information about >> the user's system could be used for many more things than just CPU >> architectures. > > Yes, the lack of extensibility is the one concern I have with baking > the CPU SSE info into the platform tag. On the other hand, the CPU > architecture info is already in there, so appending the vectorisation > support isn't an obviously bad idea, is orthogonal to the > "python.expects" consistency enforcement metadata and would cover the > NumPy use case, which is the one we really care about at this point. An extensible solution would be a big win. Maybe there should be an explicit metadata option that says "to get this piece of metadata you should install the following package and then run this command (without elevated privileges?)". Oscar From dholth at gmail.com Wed Dec 4 14:25:51 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 4 Dec 2013 08:25:51 -0500 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On Wed, Dec 4, 2013 at 6:10 AM, Nick Coghlan wrote: > There was some really good feedback in the binary dependency thread, > but it ended up going off on a few different tangents. Rather than > expecting people to read the whole thing, I figured I'd try to come up > with a summary of where it has gone so far, and where we might want to > take it from here. > > Given the diversity of topics, this should arguably be multiple new > threads, but that approach has its own problems, since some of these > topics are at least somewhat interrelated. In several cases, some of > the discussion could possible move to the tracker issues I created :) > > == Regarding documentation == > > One of the things I got from the thread is that we don't currently > have a clear overview published anywhere of *why* people use binary > extensions. The distutils docs and both dated and focus solely on the > mechanics, without discussing either use cases, or the benefits and > limitations of using them. > > We also don't have a good location to describe the approach of > statically linking or bundling additional libraries into "platlib" to > avoid issues with external dependencies and incompatible ABI changes > when distributing wheel files. > > This seems like something that would be suitably covered as an > "Advanced Topic" in the packaging user's guide, so I filed an issue > with some more specific ideas: > https://bitbucket.org/pypa/python-packaging-user-guide/issue/36/add-a-section-that-covers-binary > > > == Regarding conda == > > In terms of providing an answer to the question "Where does conda fit > in the scheme of packaging tools?", my conclusion from the thread is > that once a couple of security related issues are fixed (think PyPI > before the rubygems.org compromise for the current state of conda's > security model), and once the Python 3.3 compatibility issue is > addressed on Windows, it would be reasonable to recommend it as one of > the current options for getting hold of pre-built versions of the > scientific Python stack. > > I think this is important enough to warrant a "NumPy and the > Scientific Python stack" section in the user guide (with Linux distro > packages, Windows installers and conda all discussed as options): > > https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific > > > == Regarding alternative index servers == > > Something I believe conda supports is the idea of installer > configuration settings that are specific to a particular virtual > environment (I can't find specific docs confirming that capability, > but I certainly got that impression somewhere along the line). > > At the moment, it isn't straightforward to tell pip "when in this > virtual environment, use these additional settings", but I believe > such a feature could prove useful in dealing with some of the thornier > binary compatibility problems. In particular, it would be good to be > able to lock an environment to a particular custom index server that > it will use instead of defaulting to PyPI. > > I've posted this idea to the pip issue tracker: > https://github.com/pypa/pip/issues/1362 > > > == Regarding NumPy, build variants and API/ABI consistency == > > My current reading of the NumPy situation is that it boils down to > needing two additional features: > - a richer platform tagging mechanism (which we need for *nix systems anyway) > - a way to ensure internal consistency of the installed *builds* in an > environment, not just the abstract dependencies > > I've opened a wheel format definition issue for the first problem: > https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for > > I've posted a metadata 2.0 standard extension proposal for the latter: > https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-based-consistency-checking > (the alternative index server idea is also relevant, since PyPI > wouldn't support hosting multiple variants with the same wheel level > compatibility tags) > > > == Regarding custom installation directories == > > This technically came up in the cobblerd thread (regarding installing > scripts to /usr/sbin instead of /usr/bin), but I believe it may also > be relevant to the problem of shipping external libraries inside > wheels, static data files for applications, etc. > > It's a little underspecified in PEP 427, but the way the wheel format > currently handles installation to paths other than purelib and platlib > (or to install to both of those as part of the same wheel) is to use > the sysconfig scheme names as subdirectories within the wheel's .data > directory. This approach is great for making it easy to build > well-behaved cross platform wheels that play nice with virtual > environments, but allowing a "just put it here" escape clause could > potentially be a useful approach for platform specific wheels > (especially on *nix systems that use the Filesystem Hierarchy > Standard). > > I've posted this idea to the metadata format issue tracker: > https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to Note the 'data' sysconfig directory is already just '/' or the root of the virtualenv. From chris.barker at noaa.gov Wed Dec 4 17:05:26 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 4 Dec 2013 08:05:26 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: <1048282554706051623@unknownmsgid> Ralf, Great to have you on this thread! Note: supporting "variants" on one way or another is a great idea, but for right now, maybe we can get pretty far without it. There are options for "serious" scipy users that need optimum performance, and newbies that want the full stack. So our primary audience for "default" installs and pypi wheels are folks that need the core packages ( maybe a web dev that wants some MPL plots) and need things to "just work" more than anything optimized. So a lowest common denominator wheel would be very, very, useful. As for what that would be: the superpack is great, but it's been around a while (long while in computer years) How many non-sse machines are there still out there? How many non-sse2? And how big is the performance boost anyway? What I'm getting at is that we may well be able to build a reasonable win32 binary wheel that we can put up on pypi right now, with currently available tools. Then MPL and pandas and I python... Scipy is trickier-- what with the Fortran and all, but I think we could do Win32 anyway. And what's the hold up with win64? Is that fortran and scipy? If so, then why not do win64 for the rest of the stack? (I, for one, have been a heavy numpy user since the Numeric days, and I still hardly use scipy) By the way, we can/should do OS-X too-- it seems easier in fact (fewer hardware options to support, and the Mac's universal binaries) -Chris Note on OS-X : how long has it been since Apple shipped a 32 bit machine? Can we dump default 32 bit support? I'm pretty sure we don't need to do PPC anymore... On Dec 3, 2013, at 11:40 PM, Ralf Gommers wrote: On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: > > On Dec 3, 2013, at 7:36 PM, Oscar Benjamin > wrote: > > > On 3 December 2013 21:13, Donald Stufft wrote: > >> I think Wheels are the way forward for Python dependencies. Perhaps not > for > >> things like fortran. I hope that the scientific community can start > >> publishing wheels at least in addition too. > > > > The Fortran issue is not that complicated. Very few packages are > > affected by it. It can easily be fixed with some kind of compatibility > > tag that can be used by the small number of affected packages. > > > >> I don't believe that Conda will gain the mindshare that pip has outside > of > >> the scientific community so I hope we don't end up with two systems that > >> can't interoperate. > > > > Maybe conda won't gain mindshare outside the scientific community but > > wheel really needs to gain mindshare *within* the scientific > > community. The root of all this is numpy. It is the biggest dependency > > on PyPI, is hard to build well, and has the Fortran ABI issue. It is > > used by very many people who wouldn't consider themselves part of the > > "scientific community". For example matplotlib depends on it. The PyPy > > devs have decided that it's so crucial to the success of PyPy that > > numpy's basically being rewritten in their stdlib (along with the C > > API). > > > > A few times I've seen Paul Moore refer to numpy as the "litmus test" > > for wheels. I actually think that it's more important than that. If > > wheels are going to fly then there *needs* to be wheels for numpy. As > > long as there isn't a wheel for numpy then there will be lots of > > people looking for a non-pip/PyPI solution to their needs. > > > > One way of getting the scientific community more on board here would > > be to offer them some tangible advantages. So rather than saying "oh > > well scientific use is a special case so they should just use conda or > > something", the message should be "the wheel system provides solutions > > to many long-standing problems and is even better than conda in (at > > least) some ways because it cleanly solves the Fortran ABI issue for > > example". > > > > > > Oscar > > I?d love to get Wheels to the point they are more suitable then they are > for > SciPy stuff, That would indeed be a good step forward. I'm interested to try to help get to that point for Numpy and Scipy. I?m not sure what the diff between the current state and what > they need to be are but if someone spells it out (I?ve only just skimmed > your last email so perhaps it?s contained in that!) I?ll do the arguing > for it. I > just need someone who actually knows what?s needed to advise me :) > To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" installers for Windows containing three full builds: no SSE, SSE2 and SSE3. Plus a script that runs at install time to check which version to use. These are built with ``paver bdist_superpack``, see https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and CPU selector scripts are under tools/win32build/. How do I package those three builds into wheels and get the right one installed by ``pip install numpy``? If this is too difficult at the moment, an easier (but much less important one) would be to get the result of ``paver bdist_wininst_simple`` as a wheel. For now I think it's OK that the wheels would just target 32-bit Windows and python.org compatible Pythons (given that that's all we currently distribute). Once that works we can look at OS X and 64-bit Windows. Ralf _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Dec 4 17:41:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Dec 2013 16:41:17 +0000 Subject: [Distutils] Does pypi's simple index not set last modification time? Message-ID: I'm trying to mirror the simple index (just the index, not the referenced files) in order to do some queries on package availability. I was planning on doing a simple "wget --timestamping --recursive --level=1" but when I do, everything seems to be downloaded each time. Does PyPI not set the last modified date on the index files? If I try wget -S it looks like the last modified date is always now. Is this caused by the CDN, or is it how PyPI works? Assuming last-modified *is* inaccurate, is there any other way of doing an incremental mirror of the simple index? Paul From qwcode at gmail.com Wed Dec 4 20:03:47 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 4 Dec 2013 11:03:47 -0800 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: > - a richer platform tagging mechanism (which we need for *nix systems > anyway) > - a way to ensure internal consistency of the installed *builds* in an > environment, not just the abstract dependencies > > I've opened a wheel format definition issue for the first problem: > > https://bitbucket.org/pypa/pypi-metadata-formats/issue/15/enhance-the-platform-tag-definition-for > > I've posted a metadata 2.0 standard extension proposal for the latter: > > https://bitbucket.org/pypa/pypi-metadata-formats/issue/14/add-a-metadata-based-consistency-checking > (the alternative index server idea is also relevant, since PyPI > wouldn't support hosting multiple variants with the same wheel level > compatibility tags) > ok, but could/would the pip/wheel toolchain ever expand itself to handle delivery of external dependencies (like qt, tk, and numpy's "fortran stuff"). E.g. someone would do "pip install --tag=continuum-super-science-stack_x86_64 --index-url= http://continuum/wheels numpy and this would download a set of binary wheels: some python, some c++, some fortran, and it would all work without any OS package installs. To me, this is why Conda/Anaconda is the "easy button" for many people, because it packages and delivers common external dependencies. > > > == Regarding custom installation directories == > > This technically came up in the cobblerd thread (regarding installing > scripts to /usr/sbin instead of /usr/bin), but I believe it may also > be relevant to the problem of shipping external libraries inside > wheels, static data files for applications, etc. > > It's a little underspecified in PEP 427, but the way the wheel format > currently handles installation to paths other than purelib and platlib > (or to install to both of those as part of the same wheel) is to use > the sysconfig scheme names as subdirectories within the wheel's .data > directory. This approach is great for making it easy to build > well-behaved cross platform wheels that play nice with virtual > environments, but allowing a "just put it here" escape clause could > potentially be a useful approach for platform specific wheels > (especially on *nix systems that use the Filesystem Hierarchy > Standard). > > I've posted this idea to the metadata format issue tracker: > > https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Dec 4 20:22:15 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 4 Dec 2013 11:22:15 -0800 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: > > > "pip install --tag=continuum-super-science-stack_x86_64 --index-url= > http://continuum/wheels numpy > CORRECTION: drop the "_x86_64". that wouldn't be needed with properly packaged wheels. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Dec 4 21:56:59 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 4 Dec 2013 21:56:59 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <1048282554706051623@unknownmsgid> References: <1048282554706051623@unknownmsgid> Message-ID: On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > Ralf, > > Great to have you on this thread! > > Note: supporting "variants" on one way or another is a great idea, but for > right now, maybe we can get pretty far without it. > > There are options for "serious" scipy users that need optimum performance, > and newbies that want the full stack. > > So our primary audience for "default" installs and pypi wheels are folks > that need the core packages ( maybe a web dev that wants some MPL plots) > and need things to "just work" more than anything optimized. > The problem is explaining to people what they want - no one reads docs before grabbing a binary. On the other hand, using wheels does solve the issue that people download 32-bit installers for 64-bit Windows systems. > So a lowest common denominator wheel would be very, very, useful. > > As for what that would be: the superpack is great, but it's been around a > while (long while in computer years) > > How many non-sse machines are there still out there? How many non-sse2? > Hard to tell. Probably <2%, but that's still too much. Some older Athlon XPs don't have it for example. And what if someone submits performance optimizations (there has been a focus on those recently) to numpy that use SSE4 or AVX for example? You don't want to reject those based on the limitations of your distribution process. And how big is the performance boost anyway? > Large. For a long time we've put a non-SSE installer for numpy on pypi so that people would stop complaining that ``easy_install numpy`` didn't work. Then there were regular complaints about dot products being an order of magnitude slower than Matlab or R. What I'm getting at is that we may well be able to build a reasonable win32 > binary wheel that we can put up on pypi right now, with currently available > tools. > > Then MPL and pandas and I python... > > Scipy is trickier-- what with the Fortran and all, but I think we could do > Win32 anyway. > > And what's the hold up with win64? Is that fortran and scipy? If so, then > why not do win64 for the rest of the stack? > Yes, 64-bit MinGW + gfortran doesn't yet work (no place to install dlls from the binary, long story). A few people including David C are working on this issue right now. Visual Studio + Intel Fortran would work, but going with only an expensive toolset like that is kind of a no-go - especially since I think you'd force everyone else that builds other Fortran extensions to then also use the same toolset. (I, for one, have been a heavy numpy user since the Numeric days, and I > still hardly use scipy) > > By the way, we can/should do OS-X too-- it seems easier in fact (fewer > hardware options to support, and the Mac's universal binaries) > > -Chris > > Note on OS-X : how long has it been since Apple shipped a 32 bit machine? > Can we dump default 32 bit support? I'm pretty sure we don't need to do PPC > anymore... > I'd like to, but we decided to ship the exact same set of binaries as python.org - which means compiling on OS X 10.5/10.6 and including PPC + 32-bit Intel. Ralf > > On Dec 3, 2013, at 11:40 PM, Ralf Gommers wrote: > > > > > On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: > >> >> On Dec 3, 2013, at 7:36 PM, Oscar Benjamin >> wrote: >> >> > On 3 December 2013 21:13, Donald Stufft wrote: >> >> I think Wheels are the way forward for Python dependencies. Perhaps >> not for >> >> things like fortran. I hope that the scientific community can start >> >> publishing wheels at least in addition too. >> > >> > The Fortran issue is not that complicated. Very few packages are >> > affected by it. It can easily be fixed with some kind of compatibility >> > tag that can be used by the small number of affected packages. >> > >> >> I don't believe that Conda will gain the mindshare that pip has >> outside of >> >> the scientific community so I hope we don't end up with two systems >> that >> >> can't interoperate. >> > >> > Maybe conda won't gain mindshare outside the scientific community but >> > wheel really needs to gain mindshare *within* the scientific >> > community. The root of all this is numpy. It is the biggest dependency >> > on PyPI, is hard to build well, and has the Fortran ABI issue. It is >> > used by very many people who wouldn't consider themselves part of the >> > "scientific community". For example matplotlib depends on it. The PyPy >> > devs have decided that it's so crucial to the success of PyPy that >> > numpy's basically being rewritten in their stdlib (along with the C >> > API). >> > >> > A few times I've seen Paul Moore refer to numpy as the "litmus test" >> > for wheels. I actually think that it's more important than that. If >> > wheels are going to fly then there *needs* to be wheels for numpy. As >> > long as there isn't a wheel for numpy then there will be lots of >> > people looking for a non-pip/PyPI solution to their needs. >> > >> > One way of getting the scientific community more on board here would >> > be to offer them some tangible advantages. So rather than saying "oh >> > well scientific use is a special case so they should just use conda or >> > something", the message should be "the wheel system provides solutions >> > to many long-standing problems and is even better than conda in (at >> > least) some ways because it cleanly solves the Fortran ABI issue for >> > example". >> > >> > >> > Oscar >> >> I?d love to get Wheels to the point they are more suitable then they are >> for >> SciPy stuff, > > > That would indeed be a good step forward. I'm interested to try to help > get to that point for Numpy and Scipy. > > I?m not sure what the diff between the current state and what >> they need to be are but if someone spells it out (I?ve only just skimmed >> your last email so perhaps it?s contained in that!) I?ll do the arguing >> for it. I >> just need someone who actually knows what?s needed to advise me :) >> > > To start with, the SSE stuff. Numpy and scipy are distributed as > "superpack" installers for Windows containing three full builds: no SSE, > SSE2 and SSE3. Plus a script that runs at install time to check which > version to use. These are built with ``paver bdist_superpack``, see > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and > CPU selector scripts are under tools/win32build/. > > How do I package those three builds into wheels and get the right one > installed by ``pip install numpy``? > > If this is too difficult at the moment, an easier (but much less important > one) would be to get the result of ``paver bdist_wininst_simple`` as a > wheel. > > For now I think it's OK that the wheels would just target 32-bit Windows > and python.org compatible Pythons (given that that's all we currently > distribute). Once that works we can look at OS X and 64-bit Windows. > > Ralf > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Dec 4 22:13:52 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 4 Dec 2013 22:13:52 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Wed, Dec 4, 2013 at 9:13 AM, Paul Moore wrote: > On 4 December 2013 07:40, Ralf Gommers wrote: > >> I?m not sure what the diff between the current state and what > >> they need to be are but if someone spells it out (I?ve only just skimmed > >> your last email so perhaps it?s contained in that!) I?ll do the arguing > >> for it. I > >> just need someone who actually knows what?s needed to advise me :) > > > > > > To start with, the SSE stuff. Numpy and scipy are distributed as > "superpack" > > installers for Windows containing three full builds: no SSE, SSE2 and > SSE3. > > Plus a script that runs at install time to check which version to use. > These > > are built with ``paver bdist_superpack``, see > > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS > and > > CPU selector scripts are under tools/win32build/. > > > > How do I package those three builds into wheels and get the right one > > installed by ``pip install numpy``? > > I think that needs a compatibility tag. Certainly it isn't immediately > soluble now. > > Could you confirm how the correct one of the 3 builds is selected > (i.e., what the code is to detect which one is appropriate)? I could > look into what options we have here. > The stuff under tools/win32build I mentioned above. Specifically: https://github.com/numpy/numpy/blob/master/tools/win32build/cpuid/cpuid.c > > If this is too difficult at the moment, an easier (but much less > important > > one) would be to get the result of ``paver bdist_wininst_simple`` as a > > wheel. > > That I will certainly look into. Simple answer is "wheel convert > ". But maybe it would be worth adding a "paver bdist_wheel" > command. That should be doable in the same wahy setuptools added a > bdist_wheel command. > > > For now I think it's OK that the wheels would just target 32-bit Windows > and > > python.org compatible Pythons (given that that's all we currently > > distribute). Once that works we can look at OS X and 64-bit Windows. > > Ignoring the SSE issue, I believe that simply wheel converting > Christoph Gohlke's repository gives you that right now. The only > issues there are (1) the MKL license limitation, (2) hosting, and (3) > whether Christoph would be OK with doing this (he goes to lengths on > his site to prevent spidering his installers). > Besides the issues you mention, the problem is that it creates a single point of failure. I really appreciate everything Christoph does, but it's not appropriate as the default way to provide binary releases for a large number of projects. There needs to be a reproducible way that the devs of each project can build wheels - this includes the right metadata, but ideally also a good way to reproduce the whole build environment including compilers, blas/lapack implementations, dependencies etc. The latter part is probably out of scope for this list, but is discussed right now on the numfocus list. > I genuinely believe that "a schientific stack for non-scientists" is > trivially solved in this way. That would be nice, but no. The only thing you'd have achieved is to take a curated stack of .exe installers and converted it to the same stack of wheels. Which is nice and a step forward, but doesn't change much in the bigger picture. The problem is certainly nontrivial. Ralf > For scientists, of course, we'd need to > look deeper, but having a base to start from would be great. > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Wed Dec 4 22:29:22 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 4 Dec 2013 22:29:22 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Wed, Dec 4, 2013 at 11:41 AM, Oscar Benjamin wrote: > On 4 December 2013 07:40, Ralf Gommers wrote: > > On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: > >> > >> I?d love to get Wheels to the point they are more suitable then they are > >> for SciPy stuff, > > > > That would indeed be a good step forward. I'm interested to try to help > get > > to that point for Numpy and Scipy. > > Thanks Ralf. Please let me know what you think of the following. > > >> I?m not sure what the diff between the current state and what > >> they need to be are but if someone spells it out (I?ve only just skimmed > >> your last email so perhaps it?s contained in that!) I?ll do the arguing > >> for it. I > >> just need someone who actually knows what?s needed to advise me :) > > > > To start with, the SSE stuff. Numpy and scipy are distributed as > "superpack" > > installers for Windows containing three full builds: no SSE, SSE2 and > SSE3. > > Plus a script that runs at install time to check which version to use. > These > > are built with ``paver bdist_superpack``, see > > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS > and > > CPU selector scripts are under tools/win32build/. > > > > How do I package those three builds into wheels and get the right one > > installed by ``pip install numpy``? > > This was discussed previously on this list: > https://mail.python.org/pipermail/distutils-sig/2013-August/022362.html > Thanks, I'll go read that. Essentially the current wheel format and specification does not > provide a way to do this directly. There are several different > possible approaches. > > One possibility is that the wheel spec can be updated to include a > post-install script (I believe this will happen eventually - someone > correct me if I'm wrong). Then the numpy for Windows wheel can just do > the same as the superpack installer: ship all variants, then > delete/rename in a post-install script so that the correct variant is > in place after install. > > Another possibility is that the pip/wheel/PyPI/metadata system can be > changed to allow a "variant" field for wheels/sdists. This was also > suggested in the same thread by Nick Coghlan: > https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html > > The variant field could be used to upload multiple variants e.g. > numpy-1.7.1-cp27-cp22m-win32.whl > numpy-1.7.1-cp27-cp22m-win32-sse.whl > numpy-1.7.1-cp27-cp22m-win32-sse2.whl > numpy-1.7.1-cp27-cp22m-win32-sse3.whl > then if the user requests 'numpy:sse3' they will get the wheel with > sse3 support. > > Of course how would the user know if their CPU supports SSE3? I know > roughly what SSE is but I don't know what level of SSE is avilable on > each of the machines I use. There is a Python script/module in > numpexpr that can detect this: > https://github.com/eleddy/numexpr/blob/master/numexpr/cpuinfo.py > > When I run that script on this machine I get: > $ python cpuinfo.py > CPU information: CPUInfoBase__get_nbits=32 getNCPUs=2 has_mmx has_sse2 > is_32bit is_Core2 is_Intel is_i686 > > So perhaps someone could break that script out of numexpr and release > it as a separate package on PyPI. That's similar to what numpy has - actually it's a copy from numpy.distutils.cpuinfo > Then the instructions for installing > numpy could be something like > """ > You can install numpy with > > $pip install numpy > > which will download the default version without any CPU-specific > optimisations. > > If you know what level of SSE support your CPU has then you can > download a more optimised numpy with either of: > > $ pip install numpy:sse2 > $ pip install numpy:sse3 > > To determine whether or not your CPU has SSE2 or SSE3 or no SSE > support you can install and run the cpuinfo script. For example on > this machine: > > $ pip install cpuinfo > $ python -m cpuinfo --sse > This CPU supports the SSE3 instruction set. > > That means we can install numpy:sse3. > """ > The problem with all of the above is indeed that it's not quite automatic. You don't want your user to have to know or care about what SSE is. Nor do you want to create a new package just to hack around a pip limitation. I like the post-install (or pre-install) option much better. > Of course it would be a shame to have a solution that is so close to > automatic without quite being automatic. Also the problem is that > having no SSE support in the default numpy means that lots of people > would lose out on optimisations. For example if numpy is installed as > a dependency of something else then the user would always end up with > the unoptimised no-SSE binary. > > Another possibility is that numpy could depend on the cpuinfo package > so that it gets installed automatically before numpy. Then if the > cpuinfo package has a traditional setup.py sdist (not a wheel) it > could detect the CPU information at install time and store that in its > package metadata. Then pip would be aware of this metadata and could > use it to determine which wheel is appropriate. > > I don't quite know if this would work but perhaps the cpuinfo could > announce that it "Provides" e.g. cpuinfo:sse2. Then a numpy wheel > could "Requires" cpuinfo:sse2 or something along these lines. Or > perhaps this is better handled by the metadata extensions Nick > suggested earlier in this thread. > > I think it would be good to work out a way of doing this with e.g. a > cpuinfo package. Many other packages beyond numpy could make good use > of that metadata if it were available. Similarly having an extensible > mechanism for selecting wheels based on additional information about > the user's system could be used for many more things than just CPU > architectures. > I agree extensibility is quite important. Whatever scheme you'd think of with pre-defined tags will fail the next time anyone has a new idea (random example: what if we start shipping parallel sets of binaries that only differ in whether they're linked against ATLAS, OpenBLAS or MKL). Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Dec 4 22:59:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 4 Dec 2013 21:59:51 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 21:13, Ralf Gommers wrote: > Besides the issues you mention, the problem is that it creates a single > point of failure. I really appreciate everything Christoph does, but it's > not appropriate as the default way to provide binary releases for a large > number of projects. There needs to be a reproducible way that the devs of > each project can build wheels - this includes the right metadata, but > ideally also a good way to reproduce the whole build environment including > compilers, blas/lapack implementations, dependencies etc. The latter part is > probably out of scope for this list, but is discussed right now on the > numfocus list. You're right - what I said ignored the genuine work being done by the rest of the scientific community to solve the real issues involved. I apologise, that wasn't at all fair. Paul From ralf.gommers at gmail.com Wed Dec 4 23:10:21 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Wed, 4 Dec 2013 23:10:21 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On Wed, Dec 4, 2013 at 10:59 PM, Paul Moore wrote: > On 4 December 2013 21:13, Ralf Gommers wrote: > > Besides the issues you mention, the problem is that it creates a single > > point of failure. I really appreciate everything Christoph does, but it's > > not appropriate as the default way to provide binary releases for a large > > number of projects. There needs to be a reproducible way that the devs of > > each project can build wheels - this includes the right metadata, but > > ideally also a good way to reproduce the whole build environment > including > > compilers, blas/lapack implementations, dependencies etc. The latter > part is > > probably out of scope for this list, but is discussed right now on the > > numfocus list. > > You're right - what I said ignored the genuine work being done by the > rest of the scientific community to solve the real issues involved. I > apologise, that wasn't at all fair. > No need to apologize at all. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 5 00:31:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Dec 2013 09:31:10 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 5 Dec 2013 07:29, "Ralf Gommers" wrote: > > > > > On Wed, Dec 4, 2013 at 11:41 AM, Oscar Benjamin < oscar.j.benjamin at gmail.com> wrote: >> >> On 4 December 2013 07:40, Ralf Gommers wrote: >> > On Wed, Dec 4, 2013 at 1:54 AM, Donald Stufft wrote: >> >> >> >> I?d love to get Wheels to the point they are more suitable then they are >> >> for SciPy stuff, >> > >> > That would indeed be a good step forward. I'm interested to try to help get >> > to that point for Numpy and Scipy. >> >> Thanks Ralf. Please let me know what you think of the following. >> >> >> I?m not sure what the diff between the current state and what >> >> they need to be are but if someone spells it out (I?ve only just skimmed >> >> your last email so perhaps it?s contained in that!) I?ll do the arguing >> >> for it. I >> >> just need someone who actually knows what?s needed to advise me :) >> > >> > To start with, the SSE stuff. Numpy and scipy are distributed as "superpack" >> > installers for Windows containing three full builds: no SSE, SSE2 and SSE3. >> > Plus a script that runs at install time to check which version to use. These >> > are built with ``paver bdist_superpack``, see >> > https://github.com/numpy/numpy/blob/master/pavement.py#L224. The NSIS and >> > CPU selector scripts are under tools/win32build/. >> > >> > How do I package those three builds into wheels and get the right one >> > installed by ``pip install numpy``? >> >> This was discussed previously on this list: >> https://mail.python.org/pipermail/distutils-sig/2013-August/022362.html > > > Thanks, I'll go read that. > >> Essentially the current wheel format and specification does not >> provide a way to do this directly. There are several different >> possible approaches. >> >> One possibility is that the wheel spec can be updated to include a >> post-install script (I believe this will happen eventually - someone >> correct me if I'm wrong). Then the numpy for Windows wheel can just do >> the same as the superpack installer: ship all variants, then >> delete/rename in a post-install script so that the correct variant is >> in place after install. >> >> Another possibility is that the pip/wheel/PyPI/metadata system can be >> changed to allow a "variant" field for wheels/sdists. This was also >> suggested in the same thread by Nick Coghlan: >> https://mail.python.org/pipermail/distutils-sig/2013-August/022432.html >> >> The variant field could be used to upload multiple variants e.g. >> numpy-1.7.1-cp27-cp22m-win32.whl >> numpy-1.7.1-cp27-cp22m-win32-sse.whl >> numpy-1.7.1-cp27-cp22m-win32-sse2.whl >> numpy-1.7.1-cp27-cp22m-win32-sse3.whl >> then if the user requests 'numpy:sse3' they will get the wheel with >> sse3 support. >> >> Of course how would the user know if their CPU supports SSE3? I know >> roughly what SSE is but I don't know what level of SSE is avilable on >> each of the machines I use. There is a Python script/module in >> numpexpr that can detect this: >> https://github.com/eleddy/numexpr/blob/master/numexpr/cpuinfo.py >> >> When I run that script on this machine I get: >> $ python cpuinfo.py >> CPU information: CPUInfoBase__get_nbits=32 getNCPUs=2 has_mmx has_sse2 >> is_32bit is_Core2 is_Intel is_i686 >> >> So perhaps someone could break that script out of numexpr and release >> it as a separate package on PyPI. > > > That's similar to what numpy has - actually it's a copy from numpy.distutils.cpuinfo > >> >> Then the instructions for installing >> numpy could be something like >> """ >> You can install numpy with >> >> $pip install numpy >> >> which will download the default version without any CPU-specific optimisations. >> >> If you know what level of SSE support your CPU has then you can >> download a more optimised numpy with either of: >> >> $ pip install numpy:sse2 >> $ pip install numpy:sse3 >> >> To determine whether or not your CPU has SSE2 or SSE3 or no SSE >> support you can install and run the cpuinfo script. For example on >> this machine: >> >> $ pip install cpuinfo >> $ python -m cpuinfo --sse >> This CPU supports the SSE3 instruction set. >> >> That means we can install numpy:sse3. >> """ > > > The problem with all of the above is indeed that it's not quite automatic. You don't want your user to have to know or care about what SSE is. Nor do you want to create a new package just to hack around a pip limitation. I like the post-install (or pre-install) option much better. > >> >> Of course it would be a shame to have a solution that is so close to >> automatic without quite being automatic. Also the problem is that >> having no SSE support in the default numpy means that lots of people >> would lose out on optimisations. For example if numpy is installed as >> a dependency of something else then the user would always end up with >> the unoptimised no-SSE binary. >> >> Another possibility is that numpy could depend on the cpuinfo package >> so that it gets installed automatically before numpy. Then if the >> cpuinfo package has a traditional setup.py sdist (not a wheel) it >> could detect the CPU information at install time and store that in its >> package metadata. Then pip would be aware of this metadata and could >> use it to determine which wheel is appropriate. >> >> I don't quite know if this would work but perhaps the cpuinfo could >> announce that it "Provides" e.g. cpuinfo:sse2. Then a numpy wheel >> could "Requires" cpuinfo:sse2 or something along these lines. Or >> perhaps this is better handled by the metadata extensions Nick >> suggested earlier in this thread. >> >> I think it would be good to work out a way of doing this with e.g. a >> cpuinfo package. Many other packages beyond numpy could make good use >> of that metadata if it were available. Similarly having an extensible >> mechanism for selecting wheels based on additional information about >> the user's system could be used for many more things than just CPU >> architectures. > > > I agree extensibility is quite important. Whatever scheme you'd think of with pre-defined tags will fail the next time anyone has a new idea (random example: what if we start shipping parallel sets of binaries that only differ in whether they're linked against ATLAS, OpenBLAS or MKL). Hmm, rather than adding complexity most folks don't need directly to the base wheel spec, here's a possible "multiwheel" notion - embed multiple wheels with different names inside the multiwheel, along with a self-contained selector function for choosing which ones to actually install on the current system. This could be used not only for the NumPy use case, but also allow the distribution of external dependencies while allowing their installation to be skipped if they're already present on the target system. Cheers, Nick. > > Ralf > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Thu Dec 5 01:06:31 2013 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 4 Dec 2013 16:06:31 -0800 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: > > >> ok, but could/would the pip/wheel toolchain ever expand itself to handle >> delivery of external dependencies (like qt, tk, and numpy's "fortran >> stuff"). >> > > "fortran stuff" is pretty poorly defined -- I'm not sure we'd ever want > pip to install a fortran compiler for you.... > to be very literal, I'm talking about this anaconda "system" package http://repo.continuum.io/pkgs/free/linux-64/system-5.8-1.tar.bz2 e.g., numpy's full requirement list in anaconda is like so (specifically for numpy-1.7.1-py27_0) openssl-1.0.1c-0 python-2.7.4-0 // not re-installed when using "conda init" readline-6.2-0 sqlite-3.7.13-0 system-5.8-1 // fortran "stuff" tk-8.5.13-0 zlib-1.2.7-0 > but Anoconda does some a nifty thing: it make s conda package that holds > the shared lib, then other packages that depend on it depend on that > package, so it will both get auto--installed > But I don't see why you couldn't do that with wheels. > exactly, that's what I'm really proposing/asking, is that maybe wheels should formally go in that direction. i.e. not just packaging python projects, but packaging non-python dependencies that python projects need (but have those dependencies be optional, for those who want to fulfill those deps using the OS package mgr) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Dec 5 01:09:27 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 4 Dec 2013 16:09:27 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: On Wed, Dec 4, 2013 at 12:56 PM, Ralf Gommers wrote: > The problem is explaining to people what they want - no one reads docs > before grabbing a binary. > right -- so we want a default "pip install" install that will work for most people. And I think "works for most people" is far more important than "optimized for your system" How many non-sse machines are there still out there? How many non-sse2? >> > > Hard to tell. Probably <2%, but that's still too much. > I have no idea how to tell, but I agree 2% is too much, however, 0.2% would not be too much (IMHO) -- anyway, I'm just wondering how much we are making this hard for very little return. Anyway, best would be a select-at-runtime option -- I think that's what MKL does. IF someone can figure that out, great, but I still think a numpy wheel that works for most would still be worth doing ,and we can do it now. Some older Athlon XPs don't have it for example. And what if someone > submits performance optimizations (there has been a focus on those > recently) to numpy that use SSE4 or AVX for example? You don't want to > reject those based on the limitations of your distribution process. > No, but we also don't want to distribute nothing because we can't distribute the best thing. And how big is the performance boost anyway? >> > > Large. For a long time we've put a non-SSE installer for numpy on pypi so > that people would stop complaining that ``easy_install numpy`` didn't work. > Then there were regular complaints about dot products being an order of > magnitude slower than Matlab or R. > Does SSE by you that? or do you need a good BLAS? But same point, anyway. Though I think we lose more users by people not getting an install at all then we lose by people installing and then finding out they need a to install an optimized version to a get a good "dot". > >> Yes, 64-bit MinGW + gfortran doesn't yet work (no place to install dlls > from the binary, long story). A few people including David C are working on > this issue right now. Visual Studio + Intel Fortran would work, but going > with only an expensive toolset like that is kind of a no-go - > too bad there is no MS-fortran-express... On the other hand, saying "no one can have a 64 bit scipy, because people that want to build fortran extensions that are compatible with it are out of luck" is less than ideal. Right now, we are giving the majority of potential scipy users nothing for Win64. You know what they say "done is better than perfect" [Side note: scipy really shouldn't be a monolithic package with everything and the kitchen sink in it -- this would all be a lot easier if it was a namespace package and people could get the non-Fortran stuff by itself...but I digress.] Note on OS-X : how long has it been since Apple shipped a 32 bit machine? >> Can we dump default 32 bit support? I'm pretty sure we don't need to do PPC >> anymore... >> > > I'd like to, but we decided to ship the exact same set of binaries as > python.org - which means compiling on OS X 10.5/10.6 and including PPC + > 32-bit Intel. > no it doesn't -- if we decide not to ship the 3.9, PPC + 32-bit Intel. binary -- why should that mean that we can't ship the Intel32+64 bit one? And as for that -- if someone gets a binary with only 64 bit in it, it will run fine with the 32+64 bit build, as long as it's run on a 64 bit machine. So if, in fact, no one has a 32 bit Mac anymore (I'm not saying that's the case) we don't need to build for it. And maybe the next python.org builds could be 64 bit Intel only. Probably not yet, but we shouldn't be locked in forever.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Thu Dec 5 08:35:07 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 5 Dec 2013 08:35:07 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: On Thu, Dec 5, 2013 at 1:09 AM, Chris Barker wrote: > On Wed, Dec 4, 2013 at 12:56 PM, Ralf Gommers wrote: > >> The problem is explaining to people what they want - no one reads docs >> before grabbing a binary. >> > > right -- so we want a default "pip install" install that will work for > most people. And I think "works for most people" is far more important than > "optimized for your system" > > How many non-sse machines are there still out there? How many non-sse2? >>> >> >> Hard to tell. Probably <2%, but that's still too much. >> > > I have no idea how to tell, but I agree 2% is too much, however, 0.2% > would not be too much (IMHO) -- anyway, I'm just wondering how much we are > making this hard for very little return. > I also don't know. > Anyway, best would be a select-at-runtime option -- I think that's what > MKL does. IF someone can figure that out, great, but I still think a numpy > wheel that works for most would still be worth doing ,and we can do it now. > I'll start playing with wheels in the near future. > > Some older Athlon XPs don't have it for example. And what if someone >> submits performance optimizations (there has been a focus on those >> recently) to numpy that use SSE4 or AVX for example? You don't want to >> reject those based on the limitations of your distribution process. >> > > No, but we also don't want to distribute nothing because we can't > distribute the best thing. > > And how big is the performance boost anyway? >>> >> >> Large. For a long time we've put a non-SSE installer for numpy on pypi so >> that people would stop complaining that ``easy_install numpy`` didn't work. >> Then there were regular complaints about dot products being an order of >> magnitude slower than Matlab or R. >> > > Does SSE by you that? or do you need a good BLAS? But same point, anyway. > Though I think we lose more users by people not getting an install at all > then we lose by people installing and then finding out they need a to > install an optimized version to a get a good "dot". > > >> >>> Yes, 64-bit MinGW + gfortran doesn't yet work (no place to install dlls >> from the binary, long story). A few people including David C are working on >> this issue right now. Visual Studio + Intel Fortran would work, but going >> with only an expensive toolset like that is kind of a no-go - >> > > too bad there is no MS-fortran-express... > > On the other hand, saying "no one can have a 64 bit scipy, because people > that want to build fortran extensions that are compatible with it are out > of luck" is less than ideal. Right now, we are giving the majority of > potential scipy users nothing for Win64. > There are multiple ways to get a win64 install - Anaconda, EPD, WinPython, Christoph's installers. So there's no big hurry here. > You know what they say "done is better than perfect" > > [Side note: scipy really shouldn't be a monolithic package with everything > and the kitchen sink in it -- this would all be a lot easier if it was a > namespace package and people could get the non-Fortran stuff by > itself...but I digress.] > Namespace packages have been tried with scikits - there's a reason why scikit-learn and statsmodels spent a lot of effort dropping them. They don't work. Scipy, while monolithic, works for users. > Note on OS-X : how long has it been since Apple shipped a 32 bit >>> machine? Can we dump default 32 bit support? I'm pretty sure we don't need >>> to do PPC anymore... >>> >> >> I'd like to, but we decided to ship the exact same set of binaries as >> python.org - which means compiling on OS X 10.5/10.6 and including PPC + >> 32-bit Intel. >> > > no it doesn't -- if we decide not to ship the 3.9, PPC + 32-bit Intel. > binary -- why should that mean that we can't ship the Intel32+64 bit one? > But we do ship the 32+64-bit one (at least for Python 2.7 and 3.3). So there shouldn't be any issue here. Ralf > And as for that -- if someone gets a binary with only 64 bit in it, it > will run fine with the 32+64 bit build, as long as it's run on a 64 bit > machine. So if, in fact, no one has a 32 bit Mac anymore (I'm not saying > that's the case) we don't need to build for it. > > And maybe the next python.org builds could be 64 bit Intel only. Probably > not yet, but we shouldn't be locked in forever.... > > -Chris > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Dec 5 10:38:01 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Dec 2013 19:38:01 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: On 5 December 2013 17:35, Ralf Gommers wrote: > > Namespace packages have been tried with scikits - there's a reason why > scikit-learn and statsmodels spent a lot of effort dropping them. They don't > work. Scipy, while monolithic, works for users. The namespace package emulation that was all that was available in versions prior to 3.3 can certainly be a bit fragile at times. The native namespace packages in 3.3+ should be more robust (although even one package erroneously including an __init__.py file can still cause trouble). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Dec 5 10:40:32 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 5 Dec 2013 09:40:32 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 4 December 2013 23:31, Nick Coghlan wrote: > Hmm, rather than adding complexity most folks don't need directly to the > base wheel spec, here's a possible "multiwheel" notion - embed multiple > wheels with different names inside the multiwheel, along with a > self-contained selector function for choosing which ones to actually install > on the current system. That sounds like a reasonable approach. I'd be willing to try to put together a proof of concept implementation, if people think it's viable. What would we need to push this forward? A new PEP? > This could be used not only for the NumPy use case, but also allow the > distribution of external dependencies while allowing their installation to > be skipped if they're already present on the target system. I'm not sure how this would work - wheels don't seem to me to be appropriate for installing "external dependencies", but as I'm not 100% clear on what you mean by that term I may be misunderstanding. Can you provide a concrete example? Paul From ncoghlan at gmail.com Thu Dec 5 10:52:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Dec 2013 19:52:47 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 5 December 2013 19:40, Paul Moore wrote: > On 4 December 2013 23:31, Nick Coghlan wrote: >> Hmm, rather than adding complexity most folks don't need directly to the >> base wheel spec, here's a possible "multiwheel" notion - embed multiple >> wheels with different names inside the multiwheel, along with a >> self-contained selector function for choosing which ones to actually install >> on the current system. > > That sounds like a reasonable approach. I'd be willing to try to put > together a proof of concept implementation, if people think it's > viable. What would we need to push this forward? A new PEP? > >> This could be used not only for the NumPy use case, but also allow the >> distribution of external dependencies while allowing their installation to >> be skipped if they're already present on the target system. > > I'm not sure how this would work - wheels don't seem to me to be > appropriate for installing "external dependencies", but as I'm not > 100% clear on what you mean by that term I may be misunderstanding. > Can you provide a concrete example? If you put stuff in the "data" scheme dir, it allows you to install files anywhere you like relative to the installation root. That means you can already use the wheel format to distribute arbitrary files, you may just have to build it via some mechanism other than bdist_wheel. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Dec 5 10:54:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 5 Dec 2013 19:54:36 +1000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 4 December 2013 23:25, Daniel Holth wrote: > On Wed, Dec 4, 2013 at 6:10 AM, Nick Coghlan wrote: >> == Regarding custom installation directories == >> >> This technically came up in the cobblerd thread (regarding installing >> scripts to /usr/sbin instead of /usr/bin), but I believe it may also >> be relevant to the problem of shipping external libraries inside >> wheels, static data files for applications, etc. >> >> It's a little underspecified in PEP 427, but the way the wheel format >> currently handles installation to paths other than purelib and platlib >> (or to install to both of those as part of the same wheel) is to use >> the sysconfig scheme names as subdirectories within the wheel's .data >> directory. This approach is great for making it easy to build >> well-behaved cross platform wheels that play nice with virtual >> environments, but allowing a "just put it here" escape clause could >> potentially be a useful approach for platform specific wheels >> (especially on *nix systems that use the Filesystem Hierarchy >> Standard). >> >> I've posted this idea to the metadata format issue tracker: >> https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to > > Note the 'data' sysconfig directory is already just '/' or the root of > the virtualenv. Ah, nice - I didn't grasp that from the sysconfig docs, and it's definitely not covered in PEP 427 (it currently glosses over the install scheme directories without even a reference to sysconfig) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Thu Dec 5 11:01:28 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 5 Dec 2013 10:01:28 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: On 5 December 2013 09:52, Nick Coghlan wrote: >> I'm not sure how this would work - wheels don't seem to me to be >> appropriate for installing "external dependencies", but as I'm not >> 100% clear on what you mean by that term I may be misunderstanding. >> Can you provide a concrete example? > > If you put stuff in the "data" scheme dir, it allows you to install > files anywhere you like relative to the installation root. That means > you can already use the wheel format to distribute arbitrary files, > you may just have to build it via some mechanism other than > bdist_wheel. Ah, OK. I see. Paul From jcea at jcea.es Thu Dec 5 13:22:42 2013 From: jcea at jcea.es (Jesus Cea) Date: Thu, 05 Dec 2013 13:22:42 +0100 Subject: [Distutils] Please accept Python 3.4 selector Message-ID: <52A07012.7060602@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just publishing a new release of Berkeley DB I got "'Programming Language :: Python :: 3.4'" selector is rejected as invalid. - -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQCVAwUBUqBwEplgi5GaxT1NAQKVpgP/VHXatbfPRE0Wu9XF6psC9Ci+U/OFgHIr keD6FF+YRzpoT/hHtNWRnD+/eLfT7AigO5OXiyAhhgvwMjpv+9Q1MrtDNolpRKFl HJhyB5Po9fH6k9B/TTyngdJzRADtvfdDugWCUaxWjPwJaVH8K6ajhXo1kcE0PLig w6e6obS+Mc8= =pHa+ -----END PGP SIGNATURE----- From chris.barker at noaa.gov Thu Dec 5 18:04:54 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 5 Dec 2013 09:04:54 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: Message-ID: <-5640215006002323867@unknownmsgid> On Dec 5, 2013, at 1:40 AM, Paul Moore wrote: > > I'm not sure how this would work - wheels don't seem to me to be > appropriate for installing "external dependencies", but as I'm not > 100% clear on what you mean by that term One of the key features of conda is that it is not specifically tied to python--it can manage any binary package for a system: this is a key reason for it's existance -- continuum wants to support it's users with one way to install all they stuff they need to do their work with one cross-platform solution. This includes not just libraries that python extensions require, but also non-python stuff like Fortran compilers, other languages (like R), or who knows what? As wheels and conda packages are both just archives, there's no reason wheel couldn't grow that capability -- but I'm not at all sure we want it to. -Chris From chris.barker at noaa.gov Thu Dec 5 17:53:27 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 5 Dec 2013 08:53:27 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: <6502712414066566451@unknownmsgid> On Dec 4, 2013, at 11:35 PM, Ralf Gommers wrote I'm just wondering how much we are making this hard for very little return. I also don't know. I wonder if a poll on the relevant lists would be helpful... I'll start playing with wheels in the near future. Great! Thanks! There are multiple ways to get a win64 install - Anaconda, EPD, WinPython, Christoph's installers. So there's no big hurry here. well, this discussion is about pip-installability, but yes, some of those are python.org compatible: I know I always point people to Christoph's repo. > [Side note: scipy really shouldn't be a monolithic package with everything > and the kitchen sink in it -- this would all be a lot easier if it was a > namespace package and people could get the non-Fortran stuff by > itself...but I digress.] > Namespace packages have been tried with scikits - there's a reason why scikit-learn and statsmodels spent a lot of effort dropping them. They don't work. Scipy, while monolithic, works for users. True--I've been trying out namespace packages for some far easier problems, and you're right--not a robust solution. That really should be fixed--but a whole new topic! Note on OS-X : how long has it been since Apple shipped a 32 bit machine? >>> Can we dump default 32 bit support? I'm pretty sure we don't need to do PPC >>> anymore... >>> >> >> I'd like to, but we decided to ship the exact same set of binaries as >> python.org - which means compiling on OS X 10.5/10.6 and including PPC + >> 32-bit Intel. >> > > no it doesn't -- if we decide not to ship the 3.9, PPC + 32-bit Intel. > binary -- why should that mean that we can't ship the Intel32+64 bit one? > But we do ship the 32+64-bit one (at least for Python 2.7 and 3.3). So there shouldn't be any issue here. Right--we just need the wheel. Which should be trivial for numpy on OS-X -- not the same sse issues. Thanks for working on this. - Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at michael-jansen.biz Thu Dec 5 18:37:39 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Thu, 05 Dec 2013 18:37:39 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <2700087.3DAFl4lBSb@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2700087.3DAFl4lBSb@gambit.local.michael-jansen.biz> Message-ID: <1446874.NBV8JchrJo@gambit.local.michael-jansen.biz> On Tuesday, December 03, 2013 12:33:22 PM Michael Jansen wrote: > > Changes to distutils itself are fairly pointless, since the earliest > > possible date for publication of any such changes is now as part of > > Python 3.5 in 2015. The sheer impracticality of that approach when > > plenty of people are still running Python 2.6 is why we're going > > through the somewhat painful exercise of decoupling the preferred > > build system from the standard library :) > > > > So, at this point, the appropriate place to make such a change is in > > setuptools: https://bitbucket.org/pypa/setuptools > > > > That will allow it to be made in a way that supports Python 2.6+, > > whereas a distutils change won't really help anyone. > > A point well made :) . Will do that. I made a proof of concept implementation (not ready to be merged) here https://bitbucket.org/jansenm/setuptools/commits/all I did fight a bit with mercurial (we had severe philosophical disagreements) so i hope it works. There is a new test that checks that everything works for the distutils syntax. and then one that tests the improvements. I implemented the flexible idea. scripts = [ 'bin/bin_script', 'sbin/sbin_script', ['sbin', ['sbin/sbin_script']], ['bin', ['bin/bin_script2']], ['srv/www/myprefix', ['bin/mymod.wsgi']] The test is run with $ python setup.py test --test-module setuptools.tests.test_install_scripts; and currently prints out the directory it used for testing and does not remove it so its possible to inspect the result. test_distutils_compatibility (...) ... ###### /tmp/tmpPMKJNk I btw. noticed that numpy overloads build_scripts itself to handle functions instead of strings. Not sure if my patch will break that. Open: Where to put the scripts in step build_scripts? For now it does this for maximum backwards compatibility. /tmp/tmp5ov6qb/src/build/scripts-2.7 /tmp/tmp5ov6qb/src/build/scripts-2.7/bin_script /tmp/tmp5ov6qb/src/build/scripts-2.7/sbin_script /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7 /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/sbin /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/sbin/sbin_script /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/bin /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/bin/bin_script2 /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www/myprefix /tmp/tmp5ov6qb/src/build/setuptools-scripts-2.7/srv/www/myprefix/mymod.wsgi I had to use the install_data directory to install the scripts into with copy_tree in install_scripts because my preferred solution install_base is not rerooted by distutils (when --root is given to setup.py). There is currently no build-setuptools-scripts parameter for python setup.py build. We would have to add that. Same for build_scripts. it would be build-setuptools-dir there. Documentation is missing too. As i said only a proof of concept for now. > > Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From kamal at marimore.co.jp Thu Dec 5 20:09:11 2013 From: kamal at marimore.co.jp (Kamal Mustafa) Date: Fri, 6 Dec 2013 03:09:11 +0800 Subject: [Distutils] Buildout - redo_pyc function too slow Message-ID: Installing large package such as Django on EC2 micro instance took a very long time, 8-9 minutes with 99% cpu usage. Initially, I taught it caused by setuptools analyzing the packages to figure out it zip_safe or not [1]. But after looking at this closely, that's not the case. Analyzing the egg only took few seconds and can be negligible to the total time it took to install the whole package. I have also test by adding zip_safe=False to django's setup.py and didn't see any drastic improvement to the time taken to install it. I test by using easy_install directly and it took around 1-2 minutes to finish so it mean the other 8 minutes being spent in buildout itself rather than in setuptools/easy_install. The install process basically went like this:- ... Writing /tmp/easy_install-GwjQPW/django-master/setup.cfg Running django-master/setup.py -q bdist_egg --dist-dir /tmp/easy_install-GwjQPW/django-mas ter/egg-dist-tmp-Yk_MYR warning: no previously-included files matching '__pycache__' found under directory '*' warning: no previously-included files matching '*.py[co]' found under directory '*' ... ... Got Django 1.7. Picked: Django = 1.7 Generated script '/home/kamal/test_buildout/bin/django-admin.py'. Generated interpreter '/home/kamal/test_buildout/bin/python'. Stepping through the code, I figure out the LONG GAP starting after:- dists = self._call_easy_install( dist.location, ws, self._dest, dist) in line 531 of zc/buildout/easy_install.py. Next after this line is:- for dist in dists: redo_pyc(dist.location) Commenting this function call I manage to cut down the installation time to 2m30s. So what the purpose of this function ? Skipping it seem to be fine, I can import the package without any error. My buildout.cfg:- [buildout] parts = base [base] recipe = zc.recipe.egg eggs = Django interpreter = python The only reference to redo_pyc I found is http://www.makina-corpus.org/blog/minitage-10-out which just saying redo_pyc to be somehow slow. [1]:https://github.com/buildout/buildout/issues/116 From oscar.j.benjamin at gmail.com Thu Dec 5 22:12:17 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 5 Dec 2013 21:12:17 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: On 4 December 2013 20:56, Ralf Gommers wrote: > On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal > wrote: >> >> So a lowest common denominator wheel would be very, very, useful. >> >> As for what that would be: the superpack is great, but it's been around a >> while (long while in computer years) >> >> How many non-sse machines are there still out there? How many non-sse2? > > Hard to tell. Probably <2%, but that's still too much. Some older Athlon XPs > don't have it for example. And what if someone submits performance > optimizations (there has been a focus on those recently) to numpy that use > SSE4 or AVX for example? You don't want to reject those based on the > limitations of your distribution process. > >> And how big is the performance boost anyway? > > Large. For a long time we've put a non-SSE installer for numpy on pypi so > that people would stop complaining that ``easy_install numpy`` didn't work. > Then there were regular complaints about dot products being an order of > magnitude slower than Matlab or R. Yes, I wouldn't want that kind of bad PR getting around about scientific Python "Python is slower than Matlab" etc. It seems as if there is a need to extend the pip+wheel+PyPI system before this can fully work for numpy. I'm sure that the people here who have been working on all of this would be very interested to know what kinds of solutions would work best for numpy and related packages. You mentioned in another message that a post-install script seems best to you. I suspect there is a little reluctance to go this way because one of the goals of the wheel system is to reduce the situation where users execute arbitrary code from the internet with admin privileges e.g. "sudo pip install X" will download and run the setup.py from X with root privileges. Part of the point about wheels is that they don't need to be "executed" for installation. I know that post-install scripts are common in .deb and .rpm packages but I think that the use case there is slightly different as the files are downloaded from controlled repositories whereas PyPI has no quality assurance. BTW, how do the distros handle e.g. SSE? My understanding is that they just strip out all the SSE and related non-portable extensions and ship generic 686 binaries. My experience is with Ubuntu and I know they're not very good at handling BLAS with numpy and they don't seem to be able to compile fftpack as well as Cristoph can. Perhaps a good near-term plan might be to 1) Add the bdist_wheel command to numpy - which may actually be almost automatic with new enough setuptools/pip and wheel installed. 2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred from OS version which wheels can currently handle. 3) Upload wheels for Windows to somewhere other than PyPI e.g. SourceForge pending a distribution solution that can detect SSE support on Windows. I think it would be good to have a go at wheels even if they're not fully ready for PyPI (just in case some other issue surfaces in the process). Oscar From oscar.j.benjamin at gmail.com Thu Dec 5 22:19:58 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 5 Dec 2013 21:19:58 +0000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 5 December 2013 00:06, Marcus Smith wrote: >> >> but Anoconda does some a nifty thing: it make s conda package that holds >> the shared lib, then other packages that depend on it depend on that >> package, so it will both get auto--installed >> >> But I don't see why you couldn't do that with wheels. > > exactly, that's what I'm really proposing/asking, is that maybe wheels > should formally go in that direction. > i.e. not just packaging python projects, but packaging non-python > dependencies that python projects need (but have those dependencies be > optional, for those who want to fulfill those deps using I don't think it matters whether anyone "formally" goes in that direction. If it's possible then it will happen for some things sooner or later. I hope it does happen too, for things like build tools, BLAS/LAPACK libraries etc. Virtualenv+pip could become a much more convenient way to set up a software configuration than currently exists on Windows and OSX (and on Linux distros if you're not looking to mess with the system install). Oscar From ralf.gommers at gmail.com Thu Dec 5 23:55:43 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Thu, 5 Dec 2013 23:55:43 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: On Thu, Dec 5, 2013 at 10:12 PM, Oscar Benjamin wrote: > On 4 December 2013 20:56, Ralf Gommers wrote: > > On Wed, Dec 4, 2013 at 5:05 PM, Chris Barker - NOAA Federal > > wrote: > >> > >> So a lowest common denominator wheel would be very, very, useful. > >> > >> As for what that would be: the superpack is great, but it's been around > a > >> while (long while in computer years) > >> > >> How many non-sse machines are there still out there? How many non-sse2? > > > > Hard to tell. Probably <2%, but that's still too much. Some older Athlon > XPs > > don't have it for example. And what if someone submits performance > > optimizations (there has been a focus on those recently) to numpy that > use > > SSE4 or AVX for example? You don't want to reject those based on the > > limitations of your distribution process. > > > >> And how big is the performance boost anyway? > > > > Large. For a long time we've put a non-SSE installer for numpy on pypi so > > that people would stop complaining that ``easy_install numpy`` didn't > work. > > Then there were regular complaints about dot products being an order of > > magnitude slower than Matlab or R. > > Yes, I wouldn't want that kind of bad PR getting around about > scientific Python "Python is slower than Matlab" etc. > > It seems as if there is a need to extend the pip+wheel+PyPI system > before this can fully work for numpy. I'm sure that the people here > who have been working on all of this would be very interested to know > what kinds of solutions would work best for numpy and related > packages. > > You mentioned in another message that a post-install script seems best > to you. I suspect there is a little reluctance to go this way because > one of the goals of the wheel system is to reduce the situation where > users execute arbitrary code from the internet with admin privileges > e.g. "sudo pip install X" will download and run the setup.py from X > with root privileges. Part of the point about wheels is that they > don't need to be "executed" for installation. I know that post-install > scripts are common in .deb and .rpm packages but I think that the use > case there is slightly different as the files are downloaded from > controlled repositories whereas PyPI has no quality assurance. > I don't think it's avoidable - anything that is transparant to the user will have to execute code. The multiwheel idea of Nick looks good to me. > BTW, how do the distros handle e.g. SSE? > I don't know exactly to be honest. > My understanding is that they > just strip out all the SSE and related non-portable extensions and > ship generic 686 binaries. My experience is with Ubuntu and I know > they're not very good at handling BLAS with numpy and they don't seem > to be able to compile fftpack as well as Cristoph can. > > Perhaps a good near-term plan might be to > 1) Add the bdist_wheel command to numpy - which may actually be almost > automatic with new enough setuptools/pip and wheel installed. > 2) Upload wheels for OSX to PyPI - for OSX SSE support can be inferred > from OS version which wheels can currently handle. > 3) Upload wheels for Windows to somewhere other than PyPI e.g. > SourceForge pending a distribution solution that can detect SSE > support on Windows. > That's a reasonable plan. I have an OS X wheel already, which required only a minor change to numpy's setup.py. > I think it would be good to have a go at wheels even if they're not > fully ready for PyPI (just in case some other issue surfaces in the > process). > Agreed. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Fri Dec 6 01:02:18 2013 From: richard at python.org (Richard Jones) Date: Fri, 6 Dec 2013 11:02:18 +1100 Subject: [Distutils] Please accept Python 3.4 selector In-Reply-To: <52A07012.7060602@jcea.es> References: <52A07012.7060602@jcea.es> Message-ID: On 5 December 2013 23:22, Jesus Cea wrote: > Programming Language :: Python :: 3.4 > Added! -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 6 02:48:44 2013 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 5 Dec 2013 17:48:44 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> Message-ID: <-7116675041300409137@unknownmsgid> On Dec 5, 2013, at 1:12 PM, Oscar Benjamin wrote: > > Yes, I wouldn't want that kind of bad PR getting around about > scientific Python "Python is slower than Matlab" etc. > Well, is that better or worse that 2% or less people finding they can't run it on their old machines.... > It seems as if there is a need to extend the pip+wheel+PyPI system > before this can fully work for numpy. Maybe, in this case, but with the whole fortran ABI thing, yes. > You mentioned in another message that a post-install script seems best > to you. What would really be best is run-time selection of the appropriate lib -- it would solve this problem, and allow users to re-distribute working binaries via py2exe, etc. And not require opening a security hole in wheels... Not sure how hard that would be to do, though. > 3) Upload wheels for Windows to somewhere other than PyPI e.g. > SourceForge pending a distribution solution that can detect SSE > support on Windows. The hard-core "I want to use python instead of matlab" users are being re-directed to Anaconda or Canopy anyway. So maybe sub-optimal binaries on pypi is OK. By the way, anyone know what Anaconda and Canopy do about SSE and a good BLAS? > I think it would be good to have a go at wheels even if they're not > fully ready for PyPI (just in case some other issue surfaces in the > process). Absolutely! - Chris From donald at stufft.io Fri Dec 6 02:52:11 2013 From: donald at stufft.io (Donald Stufft) Date: Thu, 5 Dec 2013 20:52:11 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <-7116675041300409137@unknownmsgid> References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> Message-ID: <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal wrote: > What would really be best is run-time selection of the appropriate lib > -- it would solve this problem, and allow users to re-distribute > working binaries via py2exe, etc. And not require opening a security > hole in wheels... > > Not sure how hard that would be to do, though. Install time selectors probably isn?t a huge deal as long as there?s a way to force a particular variant to install and to disable the executing code. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From chris.barker at noaa.gov Fri Dec 6 05:27:57 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 5 Dec 2013 20:27:57 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Thu, Dec 5, 2013 at 5:52 PM, Donald Stufft wrote: > > On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > > > What would really be best is run-time selection of the appropriate lib > > -- it would solve this problem, and allow users to re-distribute > > working binaries via py2exe, etc. And not require opening a security > > hole in wheels... > > > > Not sure how hard that would be to do, though. > > Install time selectors probably isn?t a huge deal as long as there?s a way > to force a particular variant to install and to disable the executing code. > > I was proposing run-time -- so the same package would work "right" when moved to another machine via py2exe, etc. I imagine that's harder, particularly with permissions issues... -Chris > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 6 06:47:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Dec 2013 15:47:07 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On 6 December 2013 11:52, Donald Stufft wrote: > > On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal wrote: > >> What would really be best is run-time selection of the appropriate lib >> -- it would solve this problem, and allow users to re-distribute >> working binaries via py2exe, etc. And not require opening a security >> hole in wheels... >> >> Not sure how hard that would be to do, though. > > Install time selectors probably isn?t a huge deal as long as there?s a way > to force a particular variant to install and to disable the executing code. Hmm, I just had an idea for how to do the runtime selection thing. It actually shouldn't be that hard, so long as the numpy folks are OK with a bit of __path__ manipulation in package __init__ modules. Specifically, what could be done is this: - all of the built SSE level dependent modules would move out of their current package directories into a suitable named subdirectory (say "_nosse, _sse2, _sse3") - in the __init__.py file for each affected subpackage, you would have a snippet like: numpy._add_sse_subdir(__path__) where _add_sse_subdir would be something like: def _add_sse_subdir(search_path): if len(search_path) > 1: return # Assume the SSE dependent dir has already been added # Could likely do this SSE availability check once at import time if _have_sse3(): sub_dir = "_sse3" elif _have_sse2(): sub_dir = "_sse2" else: sub_dir = "_nosse" main_dir = search_path[0] search_path.append(os.path.join(main_dir, sub_dir) With that approach, the existing wheel model would work (no need for a variant system), and numpy installations could be freely moved between machines (or shared via a network directory). To avoid having the implicit namespace packages in 3.3+ cause any problems with this approach, the SSE subdirectories should contain __init__.py files that explicitly raise ImportError. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From theller at ctypes.org Fri Dec 6 08:10:15 2013 From: theller at ctypes.org (Thomas Heller) Date: Fri, 06 Dec 2013 08:10:15 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: <52A17857.8040104@ctypes.org> Am 06.12.2013 06:47, schrieb Nick Coghlan: > On 6 December 2013 11:52, Donald Stufft wrote: >> >> On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal wrote: >> >>> What would really be best is run-time selection of the appropriate lib >>> -- it would solve this problem, and allow users to re-distribute >>> working binaries via py2exe, etc. And not require opening a security >>> hole in wheels... >>> >>> Not sure how hard that would be to do, though. >> >> Install time selectors probably isn?t a huge deal as long as there?s a way >> to force a particular variant to install and to disable the executing code. > > Hmm, I just had an idea for how to do the runtime selection thing. It > actually shouldn't be that hard, so long as the numpy folks are OK > with a bit of __path__ manipulation in package __init__ modules. Manipulation of __path__ at runtime usually makes it harder for modulefinder to find all the required modules. Thomas From ralf.gommers at gmail.com Fri Dec 6 08:21:18 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Dec 2013 08:21:18 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan wrote: > On 6 December 2013 11:52, Donald Stufft wrote: > > > > On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > > > >> What would really be best is run-time selection of the appropriate lib > >> -- it would solve this problem, and allow users to re-distribute > >> working binaries via py2exe, etc. And not require opening a security > >> hole in wheels... > >> > >> Not sure how hard that would be to do, though. > > > > Install time selectors probably isn?t a huge deal as long as there?s a > way > > to force a particular variant to install and to disable the executing > code. > > Hmm, I just had an idea for how to do the runtime selection thing. It > actually shouldn't be that hard, so long as the numpy folks are OK > with a bit of __path__ manipulation in package __init__ modules. > > Specifically, what could be done is this: > > - all of the built SSE level dependent modules would move out of their > current package directories into a suitable named subdirectory (say > "_nosse, _sse2, _sse3") > - in the __init__.py file for each affected subpackage, you would have > a snippet like: > > numpy._add_sse_subdir(__path__) > > where _add_sse_subdir would be something like: > > def _add_sse_subdir(search_path): > if len(search_path) > 1: > return # Assume the SSE dependent dir has already been added > # Could likely do this SSE availability check once at import time > if _have_sse3(): > sub_dir = "_sse3" > elif _have_sse2(): > sub_dir = "_sse2" > else: > sub_dir = "_nosse" > main_dir = search_path[0] > search_path.append(os.path.join(main_dir, sub_dir) > > With that approach, the existing wheel model would work (no need for a > variant system), and numpy installations could be freely moved between > machines (or shared via a network directory). > Hmm, taking a compile flag and encoding it in the package layout seems like a fundamentally wrong approach. And in order to not litter the source tree and all installs with lots of empty dirs, the changes to __init__.py will have to be made at build time based on whether you're building Windows binaries or something else. Path manipulation is usually fragile as well. So I suspect this is not going to fly. Ralf > To avoid having the implicit namespace packages in 3.3+ cause any > problems with this approach, the SSE subdirectories should contain > __init__.py files that explicitly raise ImportError. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 6 13:22:59 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Dec 2013 22:22:59 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <52A17857.8040104@ctypes.org> References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> <52A17857.8040104@ctypes.org> Message-ID: On 6 December 2013 17:10, Thomas Heller wrote: > Am 06.12.2013 06:47, schrieb Nick Coghlan: >> Hmm, I just had an idea for how to do the runtime selection thing. It >> actually shouldn't be that hard, so long as the numpy folks are OK >> with a bit of __path__ manipulation in package __init__ modules. > > Manipulation of __path__ at runtime usually makes it harder for > modulefinder to find all the required modules. Not usually, always. That's why http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath exists :) However, the interesting problem in this case is that we want to package 3 different versions of the modules, choosing one of them at runtime, and modulefinder definitely *won't* cope with that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Dec 6 13:33:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Dec 2013 22:33:19 +1000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On 6 December 2013 17:21, Ralf Gommers wrote: > On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan wrote: >> With that approach, the existing wheel model would work (no need for a >> variant system), and numpy installations could be freely moved between >> machines (or shared via a network directory). > > Hmm, taking a compile flag and encoding it in the package layout seems like > a fundamentally wrong approach. And in order to not litter the source tree > and all installs with lots of empty dirs, the changes to __init__.py will > have to be made at build time based on whether you're building Windows > binaries or something else. Path manipulation is usually fragile as well. So > I suspect this is not going to fly. In the absence of the perfect solution (i.e. picking the right variant out of no SSE, SSE2, SSE3 automatically), would it be a reasonable compromise to standardise on SSE2 as "lowest acceptable common denominator"? Users with no sse capability at all or that wanted to take advantage of the SSE3 optimisations, would need to grab one of the Windows installers or something from conda, but for a lot of users, a "pip install numpy" that dropped the SSE2 version onto their system would be just fine, and a much lower barrier to entry than "well, first install this other packaging system that doesn't interoperate with your OS package manager at all...". Are we letting perfect be the enemy of better, here? (punting on the question for 6 months and seeing if we can deal with the install-time variant problem in pip 1.6 is certainly an option, but if we don't *need* to wait that long...) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Fri Dec 6 13:44:40 2013 From: donald at stufft.io (Donald Stufft) Date: Fri, 6 Dec 2013 07:44:40 -0500 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: How does conda handle SSE vs SSE2 vs SSE3? I?m digging through it?s source code and just installed numpy with it and I can?t seem to find any handling of that? On Dec 6, 2013, at 7:33 AM, Nick Coghlan wrote: > On 6 December 2013 17:21, Ralf Gommers wrote: >> On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan wrote: >>> With that approach, the existing wheel model would work (no need for a >>> variant system), and numpy installations could be freely moved between >>> machines (or shared via a network directory). >> >> Hmm, taking a compile flag and encoding it in the package layout seems like >> a fundamentally wrong approach. And in order to not litter the source tree >> and all installs with lots of empty dirs, the changes to __init__.py will >> have to be made at build time based on whether you're building Windows >> binaries or something else. Path manipulation is usually fragile as well. So >> I suspect this is not going to fly. > > In the absence of the perfect solution (i.e. picking the right variant > out of no SSE, SSE2, SSE3 automatically), would it be a reasonable > compromise to standardise on SSE2 as "lowest acceptable common > denominator"? > > Users with no sse capability at all or that wanted to take advantage > of the SSE3 optimisations, would need to grab one of the Windows > installers or something from conda, but for a lot of users, a "pip > install numpy" that dropped the SSE2 version onto their system would > be just fine, and a much lower barrier to entry than "well, first > install this other packaging system that doesn't interoperate with > your OS package manager at all...". > > Are we letting perfect be the enemy of better, here? (punting on the > question for 6 months and seeing if we can deal with the install-time > variant problem in pip 1.6 is certainly an option, but if we don't > *need* to wait that long...) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From cournape at gmail.com Fri Dec 6 14:06:07 2013 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Dec 2013 13:06:07 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 5:47 AM, Nick Coghlan wrote: > On 6 December 2013 11:52, Donald Stufft wrote: > > > > On Dec 5, 2013, at 8:48 PM, Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > > > >> What would really be best is run-time selection of the appropriate lib > >> -- it would solve this problem, and allow users to re-distribute > >> working binaries via py2exe, etc. And not require opening a security > >> hole in wheels... > >> > >> Not sure how hard that would be to do, though. > > > > Install time selectors probably isn?t a huge deal as long as there?s a > way > > to force a particular variant to install and to disable the executing > code. > > Hmm, I just had an idea for how to do the runtime selection thing. It > actually shouldn't be that hard, so long as the numpy folks are OK > with a bit of __path__ manipulation in package __init__ modules. > As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of one library, ATLAS, which as IMO the design flaw of being arch specific. I always hoped we could get away from this when I built those special installers for numpy :) MKL does not have this issue, and now that openblas (under a BSD license) can be used as well, we can alleviate this for deployment. Building a deployment story for this is not justified. David > > Specifically, what could be done is this: > > - all of the built SSE level dependent modules would move out of their > current package directories into a suitable named subdirectory (say > "_nosse, _sse2, _sse3") > - in the __init__.py file for each affected subpackage, you would have > a snippet like: > > numpy._add_sse_subdir(__path__) > > where _add_sse_subdir would be something like: > > def _add_sse_subdir(search_path): > if len(search_path) > 1: > return # Assume the SSE dependent dir has already been added > # Could likely do this SSE availability check once at import time > if _have_sse3(): > sub_dir = "_sse3" > elif _have_sse2(): > sub_dir = "_sse2" > else: > sub_dir = "_nosse" > main_dir = search_path[0] > search_path.append(os.path.join(main_dir, sub_dir) > > With that approach, the existing wheel model would work (no need for a > variant system), and numpy installations could be freely moved between > machines (or shared via a network directory). > > To avoid having the implicit namespace packages in 3.3+ cause any > problems with this approach, the SSE subdirectories should contain > __init__.py files that explicitly raise ImportError. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Dec 6 14:12:07 2013 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Dec 2013 13:12:07 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 12:44 PM, Donald Stufft wrote: > How does conda handle SSE vs SSE2 vs SSE3? I?m digging through it?s > source code and just installed numpy with it and I can?t seem to find any > handling of that? > I can't speak for conda, but @enthought, we solve it by using the MKL, which selects the right implementation at runtime. Linux distributions have system to cope with it (the hwcap capabtility of ld), but even there few packages use it. Atlas, libc are the ones I am aware of. And this breaks anyway when you use static linking obviously. David > > On Dec 6, 2013, at 7:33 AM, Nick Coghlan wrote: > > > On 6 December 2013 17:21, Ralf Gommers wrote: > >> On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan > wrote: > >>> With that approach, the existing wheel model would work (no need for a > >>> variant system), and numpy installations could be freely moved between > >>> machines (or shared via a network directory). > >> > >> Hmm, taking a compile flag and encoding it in the package layout seems > like > >> a fundamentally wrong approach. And in order to not litter the source > tree > >> and all installs with lots of empty dirs, the changes to __init__.py > will > >> have to be made at build time based on whether you're building Windows > >> binaries or something else. Path manipulation is usually fragile as > well. So > >> I suspect this is not going to fly. > > > > In the absence of the perfect solution (i.e. picking the right variant > > out of no SSE, SSE2, SSE3 automatically), would it be a reasonable > > compromise to standardise on SSE2 as "lowest acceptable common > > denominator"? > > > > Users with no sse capability at all or that wanted to take advantage > > of the SSE3 optimisations, would need to grab one of the Windows > > installers or something from conda, but for a lot of users, a "pip > > install numpy" that dropped the SSE2 version onto their system would > > be just fine, and a much lower barrier to entry than "well, first > > install this other packaging system that doesn't interoperate with > > your OS package manager at all...". > > > > Are we letting perfect be the enemy of better, here? (punting on the > > question for 6 months and seeing if we can deal with the install-time > > variant problem in pip 1.6 is certainly an option, but if we don't > > *need* to wait that long...) > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From theller at ctypes.org Fri Dec 6 14:16:17 2013 From: theller at ctypes.org (Thomas Heller) Date: Fri, 06 Dec 2013 14:16:17 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> <52A17857.8040104@ctypes.org> Message-ID: <52A1CE21.30104@ctypes.org> Am 06.12.2013 13:22, schrieb Nick Coghlan: > On 6 December 2013 17:10, Thomas Heller wrote: >> Am 06.12.2013 06:47, schrieb Nick Coghlan: >>> Hmm, I just had an idea for how to do the runtime selection thing. It >>> actually shouldn't be that hard, so long as the numpy folks are OK >>> with a bit of __path__ manipulation in package __init__ modules. >> >> Manipulation of __path__ at runtime usually makes it harder for >> modulefinder to find all the required modules. > > Not usually, always. That's why > http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath > exists :) Well, as the py2exe author and the (inactive, I admit) modulefinder maintainer I already know this. > However, the interesting problem in this case is that we want to > package 3 different versions of the modules, choosing one of them at > runtime, and modulefinder definitely *won't* cope with that. The new importlib implementation in python3.3 offers a lot a new possibilities, probably not all of them have been explored yet. For example, I have written a ModuleMapper object that, when inserted into sys.meta_path, allows transparent mapping of module names between Python2 and Python3 - no need to use six. And the new modulefinder(*) that I've written works great with that. Thomas (*) which will be part of py2exe for python3, but it is too late for python3.4. From oscar.j.benjamin at gmail.com Fri Dec 6 14:48:15 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 6 Dec 2013 13:48:15 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On 6 December 2013 13:06, David Cournapeau wrote: > > As Ralf, I think it is overkill. The problem of SSE vs non SSE is because of > one library, ATLAS, which as IMO the design flaw of being arch specific. I > always hoped we could get away from this when I built those special > installers for numpy :) > > MKL does not have this issue, and now that openblas (under a BSD license) > can be used as well, we can alleviate this for deployment. Building a > deployment story for this is not justified. Oh, okay that's great. How hard would it be to get openblas numpy wheels up and running? Would they be compatible with the existing scipy etc. binaries? Oscar From ncoghlan at gmail.com Fri Dec 6 14:54:14 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Dec 2013 23:54:14 +1000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 4 December 2013 21:10, Nick Coghlan wrote: > == Regarding conda == > > In terms of providing an answer to the question "Where does conda fit > in the scheme of packaging tools?", my conclusion from the thread is > that once a couple of security related issues are fixed (think PyPI > before the rubygems.org compromise for the current state of conda's > security model), and once the Python 3.3 compatibility issue is > addressed on Windows, it would be reasonable to recommend it as one of > the current options for getting hold of pre-built versions of the > scientific Python stack. > > I think this is important enough to warrant a "NumPy and the > Scientific Python stack" section in the user guide (with Linux distro > packages, Windows installers and conda all discussed as options): > > https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific I created a draft of this new section at https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From oscar.j.benjamin at gmail.com Fri Dec 6 15:05:23 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 6 Dec 2013 14:05:23 +0000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 6 December 2013 13:54, Nick Coghlan wrote: > On 4 December 2013 21:10, Nick Coghlan wrote: >> == Regarding conda == >> >> In terms of providing an answer to the question "Where does conda fit >> in the scheme of packaging tools?", my conclusion from the thread is >> that once a couple of security related issues are fixed (think PyPI >> before the rubygems.org compromise for the current state of conda's >> security model), and once the Python 3.3 compatibility issue is >> addressed on Windows, it would be reasonable to recommend it as one of >> the current options for getting hold of pre-built versions of the >> scientific Python stack. >> >> I think this is important enough to warrant a "NumPy and the >> Scientific Python stack" section in the user guide (with Linux distro >> packages, Windows installers and conda all discussed as options): >> >> https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific > > I created a draft of this new section at > https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff It's probably worth listing each of the full scientific Python distributions on this page (or just linking to it), rather than just Anaconda: http://www.scipy.org/install.html Oscar From ncoghlan at gmail.com Fri Dec 6 15:22:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Dec 2013 00:22:18 +1000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 7 December 2013 00:05, Oscar Benjamin wrote: > On 6 December 2013 13:54, Nick Coghlan wrote: >> On 4 December 2013 21:10, Nick Coghlan wrote: >>> I think this is important enough to warrant a "NumPy and the >>> Scientific Python stack" section in the user guide (with Linux distro >>> packages, Windows installers and conda all discussed as options): >>> >>> https://bitbucket.org/pypa/python-packaging-user-guide/issue/37/add-a-dedicated-numpy-and-the-scientific >> >> I created a draft of this new section at >> https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff > > It's probably worth listing each of the full scientific Python > distributions on this page (or just linking to it), rather than just > Anaconda: > http://www.scipy.org/install.html Done! I added a new section on "SciPy distributions" and retitled to the conda section to emphasise the open source package manager over the distribution (since the main point of that section relates to the fact that you can use conda *outside* Anaconda, even if there are still some rough edges to that approach at this point in time). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Fri Dec 6 18:11:57 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 6 Dec 2013 09:11:57 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Thu, Dec 5, 2013 at 11:21 PM, Ralf Gommers wrote: > Hmm, taking a compile flag and encoding it in the package layout seems > like a fundamentally wrong approach. > well, it's pretty ugly hack, but sometimes an ugly hack that does the job is better than nothing. IIUC, the Intel MKL libs do some sort of dynamic switching at run time too -- and that is a great feature. > And in order to not litter the source tree and all installs with lots of > empty dirs, > where "lots" what, 3? Is that so bad in a project the size of numpy? the changes to __init__.py will have to be made at build time based on > whether you're building Windows binaries or something else. > That might in fact be nicer than the "litter", but also may be a less robust and more annoying way to do it. > Path manipulation is usually fragile as well. > My first instinct was that you'd re-name directories on the fly, which might be more robust, but wouldn't work in any kind of secure environment. so a no-go. But could you elaborate on the fragile nature of sys.path manipulation? What might go wrong there? Also, it's not out of the question that once such a system was in place, that it could be used on systems other than Windows.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Dec 6 18:42:36 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Dec 2013 18:42:36 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 1:33 PM, Nick Coghlan wrote: > On 6 December 2013 17:21, Ralf Gommers wrote: > > On Fri, Dec 6, 2013 at 6:47 AM, Nick Coghlan wrote: > >> With that approach, the existing wheel model would work (no need for a > >> variant system), and numpy installations could be freely moved between > >> machines (or shared via a network directory). > > > > Hmm, taking a compile flag and encoding it in the package layout seems > like > > a fundamentally wrong approach. And in order to not litter the source > tree > > and all installs with lots of empty dirs, the changes to __init__.py will > > have to be made at build time based on whether you're building Windows > > binaries or something else. Path manipulation is usually fragile as > well. So > > I suspect this is not going to fly. > > In the absence of the perfect solution (i.e. picking the right variant > out of no SSE, SSE2, SSE3 automatically), would it be a reasonable > compromise to standardise on SSE2 as "lowest acceptable common > denominator"? > Maybe, yes. It's hard to figure out the impact of this, but I'll bring it up on the numpy list. If no one has a good way to get some statistics on cpu's that don't support these instruction sets, it may be worth a try for one of the Python versions and see how many users will run into the issue. On accident we've released an incorrect binary once before by the way (scipy 0.8.0 for Python 2.5) and that was a problem fairly quickly: https://github.com/scipy/scipy/issues/1697. That was 2010 though. > Users with no sse capability at all or that wanted to take advantage > of the SSE3 optimisations, would need to grab one of the Windows > installers or something from conda, but for a lot of users, a "pip > install numpy" that dropped the SSE2 version onto their system would > be just fine, and a much lower barrier to entry than "well, first > install this other packaging system that doesn't interoperate with > your OS package manager at all...". > Well, for most Windows users grabbing a .exe and clicking on it is a lower barrier that opening a console and typing "pip install numpy":) > Are we letting perfect be the enemy of better, here? (punting on the > question for 6 months and seeing if we can deal with the install-time > variant problem in pip 1.6 is certainly an option, but if we don't > *need* to wait that long...) > Let's first get the OS X wheels up, that can be done now. And then see what is decided on the numpy list for the compromise you propose above. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ralf.gommers at gmail.com Fri Dec 6 18:44:50 2013 From: ralf.gommers at gmail.com (Ralf Gommers) Date: Fri, 6 Dec 2013 18:44:50 +0100 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 2:48 PM, Oscar Benjamin wrote: > On 6 December 2013 13:06, David Cournapeau wrote: > > > > As Ralf, I think it is overkill. The problem of SSE vs non SSE is > because of > > one library, ATLAS, which as IMO the design flaw of being arch specific. > I > > always hoped we could get away from this when I built those special > > installers for numpy :) > > > > MKL does not have this issue, and now that openblas (under a BSD license) > > can be used as well, we can alleviate this for deployment. Building a > > deployment story for this is not justified. > > Oh, okay that's great. How hard would it be to get openblas numpy > wheels up and running? Would they be compatible with the existing > scipy etc. binaries? OpenBLAS is still pretty buggy compared to ATLAS (although performance in many cases seems to be on par); I don't think that will be well received for the official releases. We actually did discuss it as an alternative for Accelerate on OS X, but there was quite a bit of opposition. Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 6 18:44:10 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 6 Dec 2013 09:44:10 -0800 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On Fri, Dec 6, 2013 at 6:22 AM, Nick Coghlan wrote: > >> I created a draft of this new section at > >> > https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff > looks good, thanks! ONe note: """ In particular, bootstrapping conda via ``pip install conda`` and then running the ``conda init`` command provides access to all of the pre-built binaries that Continuum Analytics have created for the free version of the Anaconda distribution. """ I've been chatting off list with Travis, and while it does appear that Anaconda is more compatibly with the python.org installers than I had thought, there are still some pretty significant "rough edges", particularly with the OS-X build. But it does look pretty promising. Travis pointed out that I had been pretty critical of endorsing conda in this thread. He is right. That came from two things: 1) years of frustration with the mess of python packaging, that I thought we were finally resolving with binary wheels. 2) recent bad experience with Anaconda and teaching python to newbies. I'm pretty sure we all want "one way to do it" -- i.e. we can just tell people to install a python.org build, then use pip install to get everything else, than may simply not be practical, and apparently we're not as close as I thought. But as I think back over the last few years, I realize that I've been recommending the python.org binaries across the board, because it is the basis of a number of different package approaches. So I can tell my students: - try pip install - if that doesn't work, look in the Gohlke repo - if that doesn't work, look for a binary on the package web page. - if that doesn't work, follow the build instructions on the package web page And while it would be great if pip install always worked, this really isn't so bad, and it's not so bad because, in fact, most of the maintainers of complex packages have been targeting python.org binaries for years. So if we can't have wheels for everything, we can do pretty well if the other options (e.g. conda) are python.org compatible. That's not quite the case for Anaconda now, but it's not that far off, and Travis is interested in making it better. I'm not going to have time to poke at this myself for at least a few weeks, but at some point, maybe we can, for instance, try to convert the wxPython binaries to a conda package, or just figure out a post-install hack that will let it work with Anaconda (the current wxPython binaries for OS-X already have *.pth file magic to let both the python.org and Apple binaries use the same package). So nice work all -- it seem this is not getting as ugly as I feared. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 6 18:47:35 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 6 Dec 2013 09:47:35 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 4:33 AM, Nick Coghlan wrote: > In the absence of the perfect solution (i.e. picking the right variant > out of no SSE, SSE2, SSE3 automatically), would it be a reasonable > compromise to standardise on SSE2 as "lowest acceptable common > denominator"? > +1 > Users with no sse capability at all or that wanted to take advantage > of the SSE3 optimisations, would need to grab one of the Windows > installers or something from conda, but for a lot of users, a "pip > install numpy" that dropped the SSE2 version onto their system would > be just fine, and a much lower barrier to entry than "well, first > install this other packaging system that doesn't interoperate with > your OS package manager at all...". > exactly -- for example, I work with a web dev that could really use Matplotlib for a little task -- if I could tell him to "pip install matplotlib", he's do it, but he just sees it as too much hassle at the point... > Are we letting perfect be the enemy of better, here? I think so, yes. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 6 18:50:55 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 6 Dec 2013 09:50:55 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 5:06 AM, David Cournapeau wrote: > As Ralf, I think it is overkill. The problem of SSE vs non SSE is because > of one library, ATLAS, which as IMO the design flaw of being arch specific. > yup -- really designed for the end user to built it themselves.... > MKL does not have this issue, and now that openblas (under a BSD license) > can be used as well, we can alleviate this for deployment. Building a > deployment story for this is not justified. > So Openblas has run-time selection of the right binary? very cool! So are we done here? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Fri Dec 6 18:57:59 2013 From: cournape at gmail.com (David Cournapeau) Date: Fri, 6 Dec 2013 17:57:59 +0000 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> Message-ID: On Fri, Dec 6, 2013 at 5:50 PM, Chris Barker wrote: > On Fri, Dec 6, 2013 at 5:06 AM, David Cournapeau wrote: > >> As Ralf, I think it is overkill. The problem of SSE vs non SSE is because >> of one library, ATLAS, which as IMO the design flaw of being arch specific. >> > > yup -- really designed for the end user to built it themselves.... > > >> MKL does not have this issue, and now that openblas (under a BSD license) >> can be used as well, we can alleviate this for deployment. Building a >> deployment story for this is not justified. >> > > So Openblas has run-time selection of the right binary? very cool! So are > we done here? > Not that I know of, but you can easily build one for a given architecture, which is essentially impossible to do with Atlas reliably. I did not know about openblas instabilities, though. I guess we will have to do some more testing. David > > -Chris > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Dec 6 18:53:17 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 6 Dec 2013 09:53:17 -0800 Subject: [Distutils] Handling the binary dependency management problem In-Reply-To: <52A1CE21.30104@ctypes.org> References: <1048282554706051623@unknownmsgid> <-7116675041300409137@unknownmsgid> <91065E01-2581-45D6-8E85-75B0FB66FBA3@stufft.io> <52A17857.8040104@ctypes.org> <52A1CE21.30104@ctypes.org> Message-ID: On Fri, Dec 6, 2013 at 5:16 AM, Thomas Heller wrote: > Am 06.12.2013 13:22, schrieb Nick Coghlan: > > Manipulation of __path__ at runtime usually makes it harder for > >> modulefinder to find all the required modules. >>> >> >> Not usually, always. That's why >> http://docs.python.org/2/library/modulefinder#modulefinder.AddPackagePath >> exists :) >> > > Well, as the py2exe author and the (inactive, I admit) modulefinder > maintainer I already know this. modulefinder fails often enough that I"ve never been able ot package a non-trivial app without a bit of "force-include all of this package", (and don't-include this other thing!). So while too bad, this should not be considered deal breaker. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From kannan at cakoose.com Sat Dec 7 02:50:49 2013 From: kannan at cakoose.com (Kannan Goundan) Date: Sat, 7 Dec 2013 01:50:49 +0000 (UTC) Subject: [Distutils] More control over uploading an sdist and egg to PyPI. Message-ID: We use setuptools for our library. I'm trying to figure out a way to upload an sdist and an egg to PyPI without having to put my PyPI credentials in a file. Please let me know if this is not the right mailing list for this question. Our setup is that the entire team has access to the source code repository, but only the release managers have access to the credentials needed to make releases to PyPI (they're in a "fake" home directory). To do a release, you run: cd project-dir HOME="/release-manager-secrets/fake-pypi-home-dir" \ python setup.py bdist_egg sdist upload This works ok, but we would prefer to not have our PyPI credentials on the filesystem (unencrypted) at all. I'd like to run "setup.py bdist_egg sdist" first and then run a separate command that uploaded the egg and sdist. This command would ask for my credentials on stdin. How can I write such a program? Is there a PyPI HTTP API I can use? Does setuptools have a public interface that I can call? Thanks! - Kannan From ncoghlan at gmail.com Sat Dec 7 08:50:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Dec 2013 17:50:56 +1000 Subject: [Distutils] Binary dependency management, round 2 :) In-Reply-To: References: Message-ID: On 7 December 2013 03:44, Chris Barker wrote: > On Fri, Dec 6, 2013 at 6:22 AM, Nick Coghlan wrote: >> >> >> I created a draft of this new section at >> >> >> >> https://bitbucket.org/pypa/python-packaging-user-guide/pull-request/12/recommendations-for-numpy-et-al/diff > > looks good, thanks! With some additional suggestions from Oscar Benjamin and Ned Deilly, this is now live on the user guide: https://python-packaging-user-guide.readthedocs.org/en/latest/additional.html#installing-scientific-packages > ONe note: > > """ > In particular, bootstrapping conda via ``pip install conda`` and then > running the ``conda init`` command provides access to all of the pre-built > binaries that Continuum Analytics have created for the free version of the > Anaconda distribution. > """ > > I've been chatting off list with Travis, and while it does appear that > Anaconda is more compatibly with the python.org installers than I had > thought, there are still some pretty significant "rough edges", particularly > with the OS-X build. But it does look pretty promising. > > Travis pointed out that I had been pretty critical of endorsing conda in > this thread. He is right. That came from two things: > > 1) years of frustration with the mess of python packaging, that I thought we > were finally resolving with binary wheels. We are, but I think there's still a great deal of value in *curated* platforms. PyPI works rather well for a relatively anarchic free-for-all, but we aren't going to completely get rid of the case of somebody depending on another package that has a binary extension, but doesn't publish a wheel file until we have an integrated build farm that generates wheels from uploaded sdists automatically (such a thing is definitely on the long term wish list, but there are a great many higher priority problems ahead of it on the todo list). > 2) recent bad experience with Anaconda and teaching python to newbies. > > I'm pretty sure we all want "one way to do it" -- i.e. we can just tell > people to install a python.org build, then use pip install to get > everything else, than may simply not be practical, and apparently we're not > as close as I thought. If we can resolve the numpy situation (either with OpenBLAS, or by nominating ATLAS SSE2 as "good enough for now"), then we'll be in a pretty good place for Windows and Mac OS X. Other *nix systems will still be a challenge, since there may be more C ABI variations to cover, and the current definition of the "platform" tag doesn't do a good job of handling variation across Linux distros. > But as I think back over the last few years, I realize that I've been > recommending the python.org binaries across the board, because it is the > basis of a number of different package approaches. So I can tell my > students: > > - try pip install > > - if that doesn't work, look in the Gohlke repo > > - if that doesn't work, look for a binary on the package web page. > > - if that doesn't work, follow the build instructions on the package web > page > > And while it would be great if pip install always worked, this really isn't > so bad, and it's not so bad because, in fact, most of the maintainers of > complex packages have been targeting python.org binaries for years. > > So if we can't have wheels for everything, we can do pretty well if the > other options (e.g. conda) are python.org compatible. That's not quite the > case for Anaconda now, but it's not that far off, and Travis is interested > in making it better. The Anaconda folks just pushed a new release today which should fix the binary extension compatibility issue for 3.3 on Windows. I'm not aware of the situation on Mac OS X, but they were quite responsive to the ticket Paul filed about the compatibility problem on Windows. > I'm not going to have time to poke at this myself for at least a few weeks, > but at some point, maybe we can, for instance, try to convert the wxPython > binaries to a conda package, or just figure out a post-install hack that > will let it work with Anaconda (the current wxPython binaries for OS-X > already have *.pth file magic to let both the python.org and Apple binaries > use the same package). I believe the conda folks are also working on automatically falling back to pip - if conda doesn't find a package in the conda repos, use pip to go hunting on PyPI instead. With the PyPI package count outnumbering the curated Anaconda package list by a few orders of magnitude (~34k vs a hundred or so) such an approach should resolve a lot of "conda install doesn't work for package " issues :) > So nice work all -- it seem this is not getting as ugly as I feared. Yeah, I see a great deal of potential benefit to users in conda as a "cross platform platform", so from an upstream tooling perspective, I think it makes sense for us to treat it similarly to how we treat other platforms. The main difference is that there's additional integration in the other direction in the conda case. I think these threads have also made it clear that there's a docs issue with sysconfig and the "data" directory, but I'll start a separate thread about that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Dec 7 09:19:42 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Dec 2013 18:19:42 +1000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory Message-ID: In the binary dependency management threads, Daniel pointed out that in all currently defined sysconfig schemes, the "data" directory will end up pointing to the target installation directory. The "data" name in the scheme doesn't actually mean "this is a data file", it means "this has no defined semantic meaning at the sysconfig level, so just put it exactly where the relative path says to put it". That means that you can ship relatively arbitrary software in a wheel file by dumping it in "{distribution}-{version}.data/data/", and then installing it to the appropriate target location (e.g. if you use FHS paths inside the data directory, then you would just need to install the wheel as "pip install --root /" to get things into the desired location). However, you'd be forgiven for not realising that from the sysconfig docs - I certainly didn't realise it until Daniel pointed it out. From the sysconfig docs, I had expected "data" to mean "application data files" not "arbitrary, potentially executable, file with no assigned semantics". So, I think we should fix the docs to reflect the way the schemes are defined in practice. I'd like to clarify them to say that "data" will always point to the installation target directory for the software being installed (on POSIX systems, typically /, /usr, /usr/local, ~/.local or a virtual environment directory), and thus the "data" path is intended for any file which doesn't fit one of the predefined categories - they're arbitrary data files from the *packaging system's* point of view, but not necessarily from the *application's* point of view. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Dec 7 09:22:21 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Dec 2013 18:22:21 +1000 Subject: [Distutils] More control over uploading an sdist and egg to PyPI. In-Reply-To: References: Message-ID: On 7 December 2013 11:50, Kannan Goundan wrote: > We use setuptools for our library. I'm trying to figure out a way to upload > an sdist and an egg to PyPI without having to put my PyPI credentials in a > file. Please let me know if this is not the right mailing list for this > question. > > Our setup is that the entire team has access to the source code repository, > but only the release managers have access to the credentials needed to make > releases to PyPI (they're in a "fake" home directory). To do a release, you > run: > > cd project-dir > HOME="/release-manager-secrets/fake-pypi-home-dir" \ > python setup.py bdist_egg sdist upload > > This works ok, but we would prefer to not have our PyPI credentials on the > filesystem (unencrypted) at all. I'd like to run "setup.py bdist_egg sdist" > first and then run a separate command that uploaded the egg and sdist. This > command would ask for my credentials on stdin. > > How can I write such a program? Is there a PyPI HTTP API I can use? Does > setuptools have a public interface that I can call? Hi Kanaan, I believe twine (https://pypi.python.org/pypi/twine/) should work as the independent upload utility that you're looking for. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From info at michael-jansen.biz Sat Dec 7 11:12:10 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 11:12:10 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <1446874.NBV8JchrJo@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2700087.3DAFl4lBSb@gambit.local.michael-jansen.biz> <1446874.NBV8JchrJo@gambit.local.michael-jansen.biz> Message-ID: <5027825.GeUEDuGk4E@gambit.local.michael-jansen.biz> On Thursday, December 05, 2013 06:37:39 PM Michael Jansen wrote: > On Tuesday, December 03, 2013 12:33:22 PM Michael Jansen wrote: > > > Changes to distutils itself are fairly pointless, since the earliest > > > possible date for publication of any such changes is now as part of > > > Python 3.5 in 2015. The sheer impracticality of that approach when > > > plenty of people are still running Python 2.6 is why we're going > > > through the somewhat painful exercise of decoupling the preferred > > > build system from the standard library :) > > > > > > So, at this point, the appropriate place to make such a change is in > > > setuptools: https://bitbucket.org/pypa/setuptools > > > > > > That will allow it to be made in a way that supports Python 2.6+, > > > whereas a distutils change won't really help anyone. > > > > A point well made :) . Will do that. > > I made a proof of concept implementation (not ready to be merged) here No one interested in that? Got no answer from the maintainers too. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 11:26:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 10:26:15 +0000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: Message-ID: On 7 December 2013 08:19, Nick Coghlan wrote: > That means that you can ship relatively arbitrary software in a wheel > file by dumping it in "{distribution}-{version}.data/data/", and > then installing it to the appropriate target location (e.g. if you use > FHS paths inside the data directory, then you would just need to > install the wheel as "pip install --root /" to get things into the > desired location). > > However, you'd be forgiven for not realising that from the sysconfig > docs - I certainly didn't realise it until Daniel pointed it out. From > the sysconfig docs, I had expected "data" to mean "application data > files" not "arbitrary, potentially executable, file with no assigned > semantics". The problem with what you're proposing (which isn't a change, as you say, so much as a subtle reinterpretation) is that if you include something in a wheel under /etc or /usr/include, you *are* assigning semantics to it - and the semantics you are assigning is POSIX. That's not a disaster, of course, but can make for a less than ideal layout on Windows. For example, if packages started putting config data under "etc" that would not be normal Windows practice. It's not *bad* per se, just disconcerting (and TBH, when I've seen this done, it gives the impression of a lazily-ported Unix package). Also, this paves the way for people starting to put data files under "/usr/share" or whatever the FHS location is. That is definitely *not* the Python convention, which is to put the data inside the package and locate it via package_data(). It also makes the package fail when used zipped or via tools like py2exe. So while I have no issue with this practice as a means of dealing with things like the RPM build issue for purely POSIX packages, I do think we should be careful about raising its visibility too much. Personally, I see it as a last resort "escape hatch" for data that simply cannot be placed in any of the standard Python locations. Paul From info at michael-jansen.biz Sat Dec 7 12:02:48 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 12:02:48 +0100 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: Message-ID: <5829866.bPxD1yckDL@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 06:19:42 PM Nick Coghlan wrote: > In the binary dependency management threads, Daniel pointed out that > in all currently defined sysconfig schemes, the "data" directory will > end up pointing to the target installation directory. The "data" name > in the scheme doesn't actually mean "this is a data file", it means > "this has no defined semantic meaning at the sysconfig level, so just > put it exactly where the relative path says to put it". > > That means that you can ship relatively arbitrary software in a wheel > file by dumping it in "{distribution}-{version}.data/data/", and > then installing it to the appropriate target location (e.g. if you use > FHS paths inside the data directory, then you would just need to > install the wheel as "pip install --root /" to get things into the > desired location). No need for --root. What people use (those that have to get work done instead of discussing semantics and stuff :) ) is this: data_files = [ ('sbin/', [ ... ]), ('/etc', [....]), ('/var/lib', [...]), # I have seen this too ('../etc', [....]) That naturally (sometimes) breaks installation into prefixes != /usr. They do it because python packaging tools leave them alone on anything this is not script and package . it does not even have a way to install into the correct /usr/share/... (and windows equivalent). And i have seen people setting the install_data dir in setup.cfg to something like share/ if they only needed to install something there. To shorten their paths in setup.py. Setuptools needs fhs support. Done in a platform compatible way. Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 11:56:16 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 10:56:16 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <5027825.GeUEDuGk4E@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2700087.3DAFl4lBSb@gambit.local.michael-jansen.biz> <1446874.NBV8JchrJo@gambit.local.michael-jansen.biz> <5027825.GeUEDuGk4E@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 10:12, Michael Jansen wrote: >> I made a proof of concept implementation (not ready to be merged) here > > No one interested in that? Got no answer from the maintainers too. Personally, I'm against the idea on principle, because the bin/sbin distinction is not a cross-platform concept (Windows does not have it). For the core tools, I'd prefer *not* to include tools that encourage platform-specific behaviour (I've seen too many projects in the past which would work perfectly on Windows except for something like this). As a 3rd-party setuptools extension providing utilities that make writing POSIX-specific packages easier, I see no problem with it, though. Paul From info at michael-jansen.biz Sat Dec 7 12:10:38 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 12:10:38 +0100 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: Message-ID: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 10:26:15 AM Paul Moore wrote: > On 7 December 2013 08:19, Nick Coghlan wrote: > > That means that you can ship relatively arbitrary software in a wheel > > file by dumping it in "{distribution}-{version}.data/data/", and > > then installing it to the appropriate target location (e.g. if you use > > FHS paths inside the data directory, then you would just need to > > install the wheel as "pip install --root /" to get things into the > > desired location). > > > > However, you'd be forgiven for not realising that from the sysconfig > > docs - I certainly didn't realise it until Daniel pointed it out. From > > the sysconfig docs, I had expected "data" to mean "application data > > files" not "arbitrary, potentially executable, file with no assigned > > semantics". > > The problem with what you're proposing (which isn't a change, as you > say, so much as a subtle reinterpretation) is that if you include > something in a wheel under /etc or /usr/include, you *are* > assigning semantics to it - and the semantics you are assigning is > POSIX. That's not a disaster, of course, but can make for a less than > ideal layout on Windows. For example, if packages started putting > config data under "etc" that would not be normal Windows practice. > It's not *bad* per se, just disconcerting (and TBH, when I've seen > this done, it gives the impression of a lazily-ported Unix package). > > Also, this paves the way for people starting to put data files under > "/usr/share" or whatever the FHS location is. That is definitely > *not* the Python convention, which is to put the data inside the > package and locate it via package_data(). It also makes the package > fail when used zipped or via tools like py2exe. I guess we agree here. But people already do that because they have no other choice. Nick only wants to document the way it works better. Many people already figured this out and use it anyway. And its very hard to use data_files in a cross platform way. If you want that you are out of luck. You can't use it easily. You would have to check the platform and adapt pathes yourself. See yourself. http://code.ohloh.net/search?s=setup%20data_files&p=4&pp=0&mp=1&ml=1&me=1&md=1&ff=1& filterChecked=true Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at michael-jansen.biz Sat Dec 7 12:14:13 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 12:14:13 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <5027825.GeUEDuGk4E@gambit.local.michael-jansen.biz> Message-ID: <12273754.QCOApU5AUH@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 10:56:16 AM Paul Moore wrote: > On 7 December 2013 10:12, Michael Jansen wrote: > >> I made a proof of concept implementation (not ready to be merged) here > > > > No one interested in that? Got no answer from the maintainers too. > > Personally, I'm against the idea on principle, because the bin/sbin > distinction is not a cross-platform concept (Windows does not have > it). For the core tools, I'd prefer *not* to include tools that > encourage platform-specific behaviour (I've seen too many projects in > the past which would work perfectly on Windows except for something > like this). > > As a 3rd-party setuptools extension providing utilities that make > writing POSIX-specific packages easier, I see no problem with it, > though. My next step would be to implement support for fhs in the way that you can specify it like that '$bindir/bin_script', '$sbindir/sbin_script', ['$sbindir', ['sbin/sbin_script']], ['$bindir', ['bin/bin_script2']], ['$whateverwwwisnamedinfhs/myprefix', ['bin/mymod.wsgi']] And then setting those variables in cross platform compatible ways. Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 13:10:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 12:10:17 +0000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 11:10, Michael Jansen wrote: > I guess we agree here. But people already do that because they have no other > choice. Nick only wants to document the way it works better. Many people > already figured this out and use it anyway. I believe it's an "attractive nuisance" and should not be used except where necessary. While I'm not a fan of "security by obscurity", the current documentation limits the damage because only people who have a real need will investigate it and find out that it works. I'd rather we had a better solution that helped people do things in a cross-platform way, but in the absence of that having something that works, but isn't promoted, is a reasonable status quo. I have yet to see a use of that isn't specifically for Unix/FHS conformance. Those uses are fine, but they are *not* portable to Windows (often they work to an extent, but there are issues - e.g., they are unnatural, and they don't work in zipfile or py2exe formats). To get a standard solution, we need a proper discussion between Windows users and the various camps from the POSIX users. That's been tried before, and it's a great way of wasting many months ;-) (It also needs more Windows users than are generally available - I'm opinionated, but my use cases are atypical :-)) I don't disagree with clarifying the docs, but let's keep the wording platform-neutral, and let's not sell it as the ultimate solution, just "better than nothing, for the moment". Paul From p.f.moore at gmail.com Sat Dec 7 13:18:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 12:18:39 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <12273754.QCOApU5AUH@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <5027825.GeUEDuGk4E@gambit.local.michael-jansen.biz> <12273754.QCOApU5AUH@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 11:14, Michael Jansen wrote: > And then setting those variables in cross platform compatible ways. Well, OK, but there may not *be* a "cross platform compatible" answer - we could set sbin to the same as bin on Windows, but www is probably inside the Apache installation (or somewhere arbitrary if the user has configured things to non-default) on Windows, so there's no way Python can know this. The whole point here is that FHS gives you a way of naming things consistently, on a platform where absolue paths are the norm. On Windows, *relative* paths are the norm (often relative to the application installation directory by default) and applications' files are isolated from each other much more strongly. FHS doesn't really make sense in such an environment. But as I say, I don't have a problem with what you're doing as a POSIX solution. I just don't think we want to get into the huge cross-platform filesystem layout issue here. Just accept that it's not Windows-compatible and move on :-) Paul From info at michael-jansen.biz Sat Dec 7 13:30:24 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 13:30:24 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <12273754.QCOApU5AUH@gambit.local.michael-jansen.biz> Message-ID: <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 12:18:39 PM Paul Moore wrote: > On 7 December 2013 11:14, Michael Jansen wrote: > But as I say, I don't have a problem with what you're doing as a POSIX > solution. I just don't think we want to get into the huge > cross-platform filesystem layout issue here. Just accept that it's not > Windows-compatible and move on :-) So you consider anything that has parts that need to be installed into/under a web server as not windows compatible? Anything that wants to install pdf/html documentation? Manual Pages? Anything that needs configuration files which usually end up under /etc Anything that requires a spool file (/var/spool normally)? I am really just curious. How is this stuff solved under windows? Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 14:16:27 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 13:16:27 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <12273754.QCOApU5AUH@gambit.local.michael-jansen.biz> <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> Message-ID: I really don't have all the answers here, but... On 7 December 2013 12:30, Michael Jansen wrote: > So you consider anything that has parts that need to be installed into/under > a web server as not windows compatible? Not at all. If you're talking about documents, what's to "install"? Just ask the user to drop them under the docroot. If they are something like an Apache DLL, just ask the user in the installer where Apache is located. Or something like that. There's just no answer that assumes everyone's installation is the same and you don't have to get information from the user. > Anything that wants to install pdf/html documentation? Manual Pages? Why install them. Just put them somewhere for the user to read. Add a start menu item if you want them accessible. > Anything that needs configuration files which usually end up under /etc Well, that's a leading question. There are no such configuration files on Windows because there's no /etc. But I know what you mean and the answer is put them next to the executable. Or in the registry if you believe in that. And generally give the user a configuration app rather than expect them to manually edit a file. > Anything that requires a spool file (/var/spool normally)? I can't think of anything like that, you'd have to give an example. I'm not sure the concept exists on Windows... Of course, all of the above are *applications*, not Python modules, so I wouldn't expect you to distribute them with setuptools/PyPI anyway. But that's another aspect of the issue. Paul From info at michael-jansen.biz Sat Dec 7 14:45:32 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 14:45:32 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> Message-ID: <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 01:16:27 PM Paul Moore wrote: > I really don't have all the answers here, but... > > On 7 December 2013 12:30, Michael Jansen wrote: > > So you consider anything that has parts that need to be installed > > into/under a web server as not windows compatible? > > Not at all. If you're talking about documents, what's to "install"? > Just ask the user to drop them under the docroot. If they are > something like an Apache DLL, just ask the user in the installer where > Apache is located. Or something like that. There's just no answer that > assumes everyone's installation is the same and you don't have to get > information from the user. If we add something like $wwwroot setup.py could notice this is used and then ask the user where to put it. pip/wheel too. Currently it all gets lumped somewhere under data_files and no one has any chance to automate this. On linux the default could be to check whatever is appropriate for the distribution (/srv/www or /var/www) and if those are not there bail out with the message you have to specify the correct path on the command line. That could be the default on windows. > > > Anything that wants to install pdf/html documentation? Manual Pages? > > Why install them. Just put them somewhere for the user to read. Add a > start menu item if you want them accessible. For me installing means putting them somewhere for the user to read in an automated, reliable and reproducable fashion from sources. So building them (with sphinx or whatever) and copying somewhere for the user to read is installing. > > Anything that needs configuration files which usually end up under /etc > > Well, that's a leading question. There are no such configuration files > on Windows because there's no /etc. But I know what you mean and the > answer is put them next to the executable. Or in the registry if you > believe in that. And generally give the user a configuration app > rather than expect them to manually edit a file. But when enhancing distutils with something like $etcdir we could support that easily with minimal work for the developer. On linux install into /etc, On windows install wherever it is ok to put those stuff (without asking the user) and then provide the script author with an easy way to find out where $etcdir is on the platform so he can find his stuff. Your point of view (i don't want to think about it. Solve it posix only) means people have to develop solutions for windows separately and on top of that each and every time manually (and most won't do the work) or the other way around. A windows package will not work on linux. > > Anything that requires a spool file (/var/spool normally)? > > I can't think of anything like that, you'd have to give an example. > I'm not sure the concept exists on Windows... Thats just a file where one process pumps data in that another process reads out and handles at his leisure. > > Of course, all of the above are *applications*, not Python modules, so > I wouldn't expect you to distribute them with setuptools/PyPI anyway. > But that's another aspect of the issue. Yes it is. And i agree with you that distributing something as cobblerd with pip, pypi is not really a good idea. But setuptools is the ONLY build system that works with python code so i HAVE to use it to build and install cobblerd from sources. So it should gain support to support applications too. Not only libraries and simple scripts. Anyway. You are at least willing to discuss with me. So thanks. I guess i give up on improving setuptools and instead just implement enough to fix cobblerd. After that i guess i will take with me the hope that the next time i have contact with python packaging the situation improved but i had that hope ten years ago too :( . Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 16:15:15 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 15:15:15 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 13:45, Michael Jansen wrote: > Anyway. You are at least willing to discuss with me. So thanks. No problem! I hope I wasn't too discouraging - I get the impression that I was :-( My problem is that I don't know the answers on Windows, I only really know enough to know when something feels wrong. In this area, Unix and Windows really are very different, and it shows. Very few people are willing or able to put the effort into developing cross-platform solutions, and those that do are too busy doing it to worry about establishing general principles, etc. > I guess i give up on improving setuptools and instead just implement enough > to fix cobblerd. After that i guess i will take with me the hope that the > next time i have contact with python packaging the situation improved but i > had that hope ten years ago too :( . I can understand your situation, but please don't give up! The packaging community definitely needs people willing to help work on these issues. But it's much less about coding solutions, and more about negotiating compromises between people with very different viewpoints. To go back to your original issue (sbin vs bin) - from my point of view as a Windows developer, packages just export commands. Why would I choose to put them in the sbin list rather than the bin list? If that's not clear, then I'll possibly develop packages that don't work "right" on Unix. Let's thrash out the answer to that question first, and *then* worry about implementing it. Is the relevant point that "bin" is on PATH but sbin (typically) isn't? If both are on PATH, I don't see why you bother having two directories. So maybe the real idea here is to split entry points into "commands that should be on PATH" and "commands that should be provided, but not on PATH by default". Or is it? My understanding of sbin on Unix is that it's somehow "system utilities" - but that's just a classification with no practical impact. (What makes formatting a disk a "system utility"? I used to do it all the time when floppy disks were common.) Or maybe sbin is only on PATH when you're root. How does that translate to typical Windows systems, where there is usually only one user, and he has access to everything? Maybe in that context, sbin should be for utilities that require elevation? Lots of questions, but no obvious answers... As I said, and I'll say again, you don't *have* to answer these questions. But not doing so is not "I don't want to think about it, just solve POSIX". Rather it's "let's not make everything wait for a perfect solution when POSIX people need something now". But equally, let's not stop working on the cross-platform solution because the POSIX people have something and no longer care - that's *why* the problem never gets solved properly. One other thing I should say here - you mention your project is "cobblerd". The "d" on the end makes me thing it's a daemon. That's inherently POSIX only - on Windows this would be a service (which is a whole other beast). Actually looking at Cobbler on PyPI it says "Cobbler is a Linux boot server..." so Windows compatibility really isn't crucial in this particular instance, I guess :-) Paul From ncoghlan at gmail.com Sat Dec 7 16:40:10 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Dec 2013 01:40:10 +1000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: As part of the docs clarification, I was planning to point out that such layouts are almost always platform (and even distro) specific. That's still a valid use case for sdist and wheel, though, so it makes sense to me to document it properly. My other related question is whether it's possible to force bdist_wheel to include the platform tag as part of the build process, since such wheels *should* be flagged as platform specific. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 7 16:53:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Dec 2013 01:53:29 +1000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <2345586.xRyqFqWx8P@gambit.local.michael-jansen.biz> <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> Message-ID: FWIW, tackling FHS compliance is one of the things I have in mind as being suitable for a metadata 2.0 extension. Because Mac OS X and Windows just bundle (almost) everything into the application directory, FHS concepts simply don't map properly, so the current best FHS compliant solution is to use the "data" scheme path, FHS subdirectories and flag the project as Linux specific in the docs (or just not publish it on PyPI). Alternatively, cross platform Python code sometimes just ignores FHS compliance and bundles everything in with the module code (using pkgutil.get_data to retrieve it at runtime). Getting such packages ready for distro inclusion then ends up being quite a pain :P Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From info at michael-jansen.biz Sat Dec 7 17:56:10 2013 From: info at michael-jansen.biz (Michael Jansen) Date: Sat, 07 Dec 2013 17:56:10 +0100 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> Message-ID: <1438578.gDLamZSRKg@gambit.local.michael-jansen.biz> On Saturday, December 07, 2013 03:15:15 PM Paul Moore wrote: > On 7 December 2013 13:45, Michael Jansen wrote: > I can understand your situation, but please don't give up! The > packaging community definitely needs people willing to help work on > these issues. But it's much less about coding solutions, and more > about negotiating compromises between people with very different > viewpoints. And thats where i disagree. I have some experience in open source contributions (KDE, cobbler and my own). 1. Usable code always wins. Always. Ideas and Agreements solve nothing. 2. This is a already solved problem. Both autotools and even more so cmake make it possible to develop cross platform. Why python should be differently is something i won't understand. If you want to wait for a compromise and an agreement nothing will ever be implemented. > To go back to your original issue (sbin vs bin) - from my point of > view as a Windows developer, packages just export commands. Why would > I choose to put them in the sbin list rather than the bin list? If > that's not clear, then I'll possibly develop packages that don't work > "right" on Unix. Let's thrash out the answer to that question first, > and *then* worry about implementing it. Is the relevant point that > "bin" is on PATH but sbin (typically) isn't? If both are on PATH, I > don't see why you bother having two directories. So maybe the real > idea here is to split entry points into "commands that should be on > PATH" and "commands that should be provided, but not on PATH by > default". Or is it? My understanding of sbin on Unix is that it's > somehow "system utilities" - but that's just a classification with no > practical impact. (What makes formatting a disk a "system utility"? I > used to do it all the time when floppy disks were common.) Or maybe > sbin is only on PATH when you're root. How does that translate to > typical Windows systems, where there is usually only one user, and he > has access to everything? Maybe in that context, sbin should be for > utilities that require elevation? Lots of questions, but no obvious > answers... A windows developer should not care about the difference. Both end up in the same directory. A unix developer will either not care too (and be reminded by the distro) or has read http://www.pathname.com/fhs/pub/fhs-2.3.html#SBINSYSTEMBINARIES So the not caring unix/windows developer will one day get a bug report/patch from a unix user or distro packager if some script belongs to sbin. > As I said, and I'll say again, you don't *have* to answer these > questions. But not doing so is not "I don't want to think about it, > just solve POSIX". Rather it's "let's not make everything wait for a > perfect solution when POSIX people need something now". But equally, > let's not stop working on the cross-platform solution because the > POSIX people have something and no longer care - that's *why* the > problem never gets solved properly. Thats why i prefer to improve step by step. The sbin/bin distinction can be solved in a platform neutral way so why not do it. > One other thing I should say here - you mention your project is > "cobblerd". The "d" on the end makes me thing it's a daemon. That's > inherently POSIX only - on Windows this would be a service (which is a > whole other beast). Actually looking at Cobbler on PyPI it says > "Cobbler is a Linux boot server..." so Windows compatibility really > isn't crucial in this particular instance, I guess :-) It most likely will never run on windows but i have seen weirder things (productive clearcase running on windows xp *shudder*). But there is nothing that prevents it from ever running on windows. So if i solved the multi distro support there is nothing that prevents someone to make it running under windows. Why shouldn't you boot and provision your unix virtual machines with a server running on windows? But i guess its unlikely. And its not my project. I only contribute since about three weeks. My homepage says something about "drive-by coding" and thats what i do. I like the project and noticed it currently only runs on redhat and installed into /usr. There are people using it on sles (its a rather enterprisey thing) which use patches. So i solved all obstacles to get it running out of the box (with correct paths) on opensuse already and thought i could do another attempt to get those improvements that make sense into setuptools. But i think that won't happen. And it was not you. I like to discuss and i always applaud people that help other brainstorm. Anyway. I will solve my problems on top of setuptools for cobblerd only for now. Then i will think about what to do with the results. I need and partially have solution for: - configure some files (patch @@var@@) (implemented) - eg. patch VIRTUALENV into a apache config file for wsgi virtualenv support - patch some distro / platform specific paths in config files. - install scripts in locations != bin (sbin and some wsgi) (implemented) - support for share/ (not yet) - support for /etc (different for prefix=/usr|/usr/local) (not yet) - support for /var (not yet) - support for /srv/www (sometimes its /var/www (done) Currently all those pathes are just done with data_files which makes multi distro support quite complicated there. Yes this is mostly linux distro stuff but some of it could be useful for windows too. Mike -- Michael Jansen http://michael-jansen.biz -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 18:41:05 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 17:41:05 +0000 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: <1438578.gDLamZSRKg@gambit.local.michael-jansen.biz> References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> <1438578.gDLamZSRKg@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 16:56, Michael Jansen wrote: > A windows developer should not care about the difference. Both end up in the > same directory. [...] will one day get a bug report/patch from a unix user OK, cool. That makes sense. Paul From jaraco at jaraco.com Sat Dec 7 19:29:22 2013 From: jaraco at jaraco.com (Jason R. Coombs) Date: Sat, 7 Dec 2013 18:29:22 +0000 Subject: [Distutils] Setuptools 2.0 Released Message-ID: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> The PyPA is pleased to announce the Setuptools 2.0 release. This backward-incompatible release drops support for Python 2.4 and Python 2.5, but is otherwise compatible with 1.x releases. The 1.x series will continue to be supported for bug and security fixes for the foreseeable future. This means that any systems with older versions of Python should install "setuptools<2dev" or use the legacy bootstrap script, available at https://bitbucket.org/pypa/setuptools/raw/bootstrap-py24/ez_setup.py for Python 2.4 and Python 2.5 users. That link is also published in the README as install instructions. Please report any emergent issues at https://bitbucket.org/pypa/setuptools/issues. Jason PyPA -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Dec 7 21:48:27 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 7 Dec 2013 12:48:27 -0800 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: > As part of the docs clarification, I was planning to point out that such > layouts are almost always platform (and even distro) specific. That's still > a valid use case for sdist and wheel, though, so it makes sense to me to > document it properly. > > My other related question is whether it's possible to force bdist_wheel to > include the platform tag as part of the build process, since such wheels > *should* be flagged as platform specific. > that would be nice at least for absolute paths (like "/etc"), or relative paths that go up the folder hierarchy into who knows where. for non-upward relative paths, it's less clear whether it should be marked platform specific -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Dec 7 22:32:35 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 7 Dec 2013 13:32:35 -0800 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: actually, bdist_wheel doesn't handle absolute paths in the "data_files" keyword like standard setuptools installs do, which honor it as absolute (which seems to match the examples in the distutils docs) when using absolute paths, the data ends up in the packaged wheel at the top level (not in the {distribution}-{version}.data folder). And get's installed relative to site-packages (along with the project packages) so, bdist_wheel is re-interpreting distutil's "data_files" differently. maybe better for wheel to fail to build projects with absolute data_files, than to just reinterpret it like that. On Sat, Dec 7, 2013 at 12:48 PM, Marcus Smith wrote: > > As part of the docs clarification, I was planning to point out that such >> layouts are almost always platform (and even distro) specific. That's still >> a valid use case for sdist and wheel, though, so it makes sense to me to >> document it properly. >> >> My other related question is whether it's possible to force bdist_wheel >> to include the platform tag as part of the build process, since such wheels >> *should* be flagged as platform specific. >> > that would be nice at least for absolute paths (like "/etc"), or relative > paths that go up the folder hierarchy into who knows where. > for non-upward relative paths, it's less clear whether it should be marked > platform specific > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Sat Dec 7 22:44:53 2013 From: qwcode at gmail.com (Marcus Smith) Date: Sat, 7 Dec 2013 13:44:53 -0800 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files On Sat, Dec 7, 2013 at 1:32 PM, Marcus Smith wrote: > actually, bdist_wheel doesn't handle absolute paths in the "data_files" > keyword like standard setuptools installs do, which honor it as absolute > (which seems to match the examples in the distutils docs) > when using absolute paths, the data ends up in the packaged wheel at the > top level (not in the {distribution}-{version}.data folder). > And get's installed relative to site-packages (along with the project > packages) > > so, bdist_wheel is re-interpreting distutil's "data_files" differently. > maybe better for wheel to fail to build projects with absolute data_files, > than to just reinterpret it like that. > > > > > > > On Sat, Dec 7, 2013 at 12:48 PM, Marcus Smith wrote: > >> >> As part of the docs clarification, I was planning to point out that such >>> layouts are almost always platform (and even distro) specific. That's still >>> a valid use case for sdist and wheel, though, so it makes sense to me to >>> document it properly. >>> >>> My other related question is whether it's possible to force bdist_wheel >>> to include the platform tag as part of the build process, since such wheels >>> *should* be flagged as platform specific. >>> >> that would be nice at least for absolute paths (like "/etc"), or relative >> paths that go up the folder hierarchy into who knows where. >> for non-upward relative paths, it's less clear whether it should be >> marked platform specific >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 7 23:22:44 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Dec 2013 22:22:44 +0000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: On 7 December 2013 21:32, Marcus Smith wrote: > actually, bdist_wheel doesn't handle absolute paths in the "data_files" > keyword like standard setuptools installs do, which honor it as absolute > (which seems to match the examples in the distutils docs) > when using absolute paths, the data ends up in the packaged wheel at the top > level (not in the {distribution}-{version}.data folder). > And get's installed relative to site-packages (along with the project > packages) > > so, bdist_wheel is re-interpreting distutil's "data_files" differently. > maybe better for wheel to fail to build projects with absolute data_files, > than to just reinterpret it like that. +1. It seems to me (from a Windows user's perspective) that projects that use absolute paths (or relative paths that go above the installation root) are fundamentally different from ones that don't. For a start they feel more like "applications" than "Python packages". Maybe what we want is a sysconfig path "root" that explicitly encodes the filesystem root. Then we can require that *all* paths are relative to some sysconfig path, and must not use ".." to go "up" from there. Projects that use the "root" path can be clearly noted as not wheel-compatible (and quite likely not cross-platform). By making it an explicit path, we can make the policy explicit and provide better error messages and metadata. Paul PS Please note that as a Windows user, I don't have any practical experience of projects with requirements for absolue paths - they are alien to the Windows world. So my views should be taken in that light (i.e, as being of limited value :-)). From chris.barker at noaa.gov Sat Dec 7 23:50:04 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 7 Dec 2013 14:50:04 -0800 Subject: [Distutils] Install a script to /sbin instead of /bin In-Reply-To: References: <1459368.Klcf8yIKRW@gambit.local.michael-jansen.biz> <7140896.QqBLCepriC@gambit.local.michael-jansen.biz> <1438578.gDLamZSRKg@gambit.local.michael-jansen.biz> Message-ID: Just a note here: the wxWidgets (and thus wxPython, natch) project has a wxStandardPaths object: http://docs.wxwidgets.org/trunk/classwx_standard_paths.html It provides a cross platform way to get, well, the standard paths an application might need: GetAppDocumentsDir () GetConfigDir () GetDataDir () GetDocumentsDir () GetExecutablePath () GetInstallPrefix () GetLocalDataDir () GetLocalizedResourcesDir () GetPluginsDir () GetResourcesDir () GetTempDir () GetUserConfigDir () GetUserDataDir () GetUserLocalDataDir () ..... These all "do the right thing" on the supported platforms -- something may be in the application install dir on Windows, int he app bundle on the MAc, and it /etc on Linux, for instance. granted, wx is an application framework, so it needs this stuff, but while pip is about installing python packages, there really is no clear line between at package with a script or two and an application, and even simple scripts may need a few of these things. I don't see why we couldn't include a set like this, and have platform-specific mappings. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From kannan at cakoose.com Sun Dec 8 00:25:49 2013 From: kannan at cakoose.com (Kannan Goundan) Date: Sat, 7 Dec 2013 23:25:49 +0000 (UTC) Subject: [Distutils] =?utf-8?q?More_control_over_uploading_an_sdist_and_eg?= =?utf-8?q?g_to=09PyPI=2E?= References: Message-ID: Nick Coghlan gmail.com> writes: > Hi Kanaan, > > I believe twine (https://pypi.python.org/pypi/twine/) should work as > the independent upload utility that you're looking for. > > Cheers, > Nick. That's perfect, thanks. (Also thanks to ??Daniel Holth who replied off-list.) Another great thing about Twine is that it lets you do the GPG signing yourself. This is important for us since our release signing key isn't in each user's keyring but it is in a separate keyring file. From ncoghlan at gmail.com Sun Dec 8 03:01:33 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Dec 2013 12:01:33 +1000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: On 8 December 2013 08:22, Paul Moore wrote: > On 7 December 2013 21:32, Marcus Smith wrote: >> actually, bdist_wheel doesn't handle absolute paths in the "data_files" >> keyword like standard setuptools installs do, which honor it as absolute >> (which seems to match the examples in the distutils docs) >> when using absolute paths, the data ends up in the packaged wheel at the top >> level (not in the {distribution}-{version}.data folder). >> And get's installed relative to site-packages (along with the project >> packages) >> >> so, bdist_wheel is re-interpreting distutil's "data_files" differently. >> maybe better for wheel to fail to build projects with absolute data_files, >> than to just reinterpret it like that. > > +1. It seems to me (from a Windows user's perspective) that projects > that use absolute paths (or relative paths that go above the > installation root) are fundamentally different from ones that don't. > For a start they feel more like "applications" than "Python packages". This started with my suggestion to create a separate directory in in the wheel format for bundling arbitrary other files: https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to Daniel initially suggested that such a thing would be redundant with the existing "data" directory in sysconfig, but given the subsequent discussions, I no longer think that is true. A new scheme directory would be a simpler solution than changing the wheel format, though. > Maybe what we want is a sysconfig path "root" that explicitly encodes > the filesystem root. s/filesystem root/installation root/ I'm assuming that's what you meant, since you'd know better than I do that "filesystem root" isn't a particularly useful concept on Windows :) I guess "drive root" would be another way to interpret it, but that doesn't seem useful either. > Then we can require that *all* paths are relative > to some sysconfig path, and must not use ".." to go "up" from there. > Projects that use the "root" path can be clearly noted as not > wheel-compatible (and quite likely not cross-platform). Well, they'd be wheel compatible - the wheels would just need to be marked as platform specific. > By making it an explicit path, we can make the policy explicit and > provide better error messages and metadata. Agreed, it sounds like "data" is currently being used to mean two different things, and we need to define a new scheme directory to separate out the FHS compliance use case. > Paul > > PS Please note that as a Windows user, I don't have any practical > experience of projects with requirements for absolue paths - they are > alien to the Windows world. So my views should be taken in that light > (i.e, as being of limited value :-)). Yeah, absolute paths are mostly about playing nice with FHS, which is mostly about playing nice with standardised system administration tools. For a server: - /var/* is the mutable data you need to back up - /etc/* is the config to be controlled by your config management system - most everything else should be controlled by the system package manager Distro packagers and system administrators get grumpy when upstream software doesn't play by those rules, because it makes things harder to audit, harder to back up, and just generally harder to administer. Perhaps rather than trying to be generic with a name like "root", we could just define an explicit "fhs" path? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Dec 8 09:52:59 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 8 Dec 2013 08:52:59 +0000 Subject: [Distutils] Clarifying the meaning of the sysconfig "data" directory In-Reply-To: References: <4903588.L2m8R8uuFQ@gambit.local.michael-jansen.biz> Message-ID: On 8 December 2013 02:01, Nick Coghlan wrote: >> Maybe what we want is a sysconfig path "root" that explicitly encodes >> the filesystem root. > > s/filesystem root/installation root/ > > I'm assuming that's what you meant, since you'd know better than I do > that "filesystem root" isn't a particularly useful concept on Windows > :) I actually did mean "filesystem root" - precisely to make the point that it isn't useful on Windows. The installation root is already covered by "data". Your suggestion of "fhs" makes that even more explicit. Paul From holger at merlinux.eu Mon Dec 9 13:39:09 2013 From: holger at merlinux.eu (holger krekel) Date: Mon, 9 Dec 2013 12:39:09 +0000 Subject: [Distutils] PyPI download issues from Rackspace Cloud In-Reply-To: <20131125064805.GA19537@fridge.pov.lt> References: <20131123100536.GA30416@fridge.pov.lt> <20131125064805.GA19537@fridge.pov.lt> Message-ID: <20131209123909.GU7196@merlinux.eu> On Mon, Nov 25, 2013 at 08:48 +0200, Marius Gedminas wrote: > On Sat, Nov 23, 2013 at 08:59:35AM -0500, Donald Stufft wrote: > > Can you try with 1.5rc1? > > This was trickier than I thought, because pip appears to be incapable of > upgrading itself on Windows: > > $ git clone https://github.com/pypa/pip > $ virtualenv env > $ env/scripts/pip install -U ./pip > WindowsError: [Error 5] Access is denied: 'c:\\users\\mg\\...\\scripts\\pip.exe' > > but the advice in https://github.com/pypa/pip/issues/1299 helped: > > $ env/scripts/python -m pip -U ./pip > > > We switched to requests in that version and perhaps it side steps the issue? > > Looks like. 10 out of 10 successful downloads with pip git master using: > > $ unset PIP_DOWNLOAD_CACHE > $ rm -f virtualenv-1.10.1.tar.gz && env/scripts/pip install -d . virtualenv > > As a control, I tested again with pip 1.4.1, and 2 out of 4 downloads > failed with a bad hash error. > > Interesting. Thanks for the notes, Marius. I also got md5 checksumming issues with pip-1.4.1 on a windows machine in the rackspace cloud. Alas, i cannot easily force a different pip version because pip is installed from an automation script which comes from somewhere else :( best, holger From erik.m.bray at gmail.com Mon Dec 9 21:06:16 2013 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 9 Dec 2013 15:06:16 -0500 Subject: [Distutils] Setuptools 2.0 Released In-Reply-To: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> References: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> Message-ID: On Sat, Dec 7, 2013 at 1:29 PM, Jason R. Coombs wrote: > The PyPA is pleased to announce the Setuptools 2.0 release. > > > > This backward-incompatible release drops support for Python 2.4 and Python > 2.5, but is otherwise compatible with 1.x releases. > > > > The 1.x series will continue to be supported for bug and security fixes for > the foreseeable future. This means that any systems with older versions of > Python should install ?setuptools<2dev? or use the legacy bootstrap script, > available at > https://bitbucket.org/pypa/setuptools/raw/bootstrap-py24/ez_setup.py for > Python 2.4 and Python 2.5 users. That link is also published in the README > as install instructions. > > > > Please report any emergent issues at > https://bitbucket.org/pypa/setuptools/issues. Thanks for the awesome work! I still support 2.5 for a few projects so it's helpful to know that there are still options available for such cases. Erik From solipsis at pitrou.net Mon Dec 9 22:40:34 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 9 Dec 2013 21:40:34 +0000 (UTC) Subject: [Distutils] Setuptools 2.0 Released References: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> Message-ID: Hi, Jason R. Coombs jaraco.com> writes: > > The PyPA is pleased to announce the Setuptools 2.0 release. > ? > This backward-incompatible release drops support for Python 2.4 and Python 2.5, but is otherwise compatible with 1.x releases. Is there a changelog? Regards Antoine. From tseaver at palladion.com Mon Dec 9 23:03:25 2013 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 09 Dec 2013 17:03:25 -0500 Subject: [Distutils] Setuptools 2.0 Released In-Reply-To: References: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/09/2013 04:40 PM, Antoine Pitrou wrote: > Is there a changelog? https://bitbucket.org/pypa/setuptools/src/f4d6757ae9fa79df03ef6ed070ab8757707fe963/CHANGES.txt?at=default Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlKmPi0ACgkQ+gerLs4ltQ6HggCguPzkufm/VmuORsxC5xJ59rwQ yY4An2y622rRFmNFrNOxEit1TED0QpMw =P0Ix -----END PGP SIGNATURE----- From ncoghlan at gmail.com Tue Dec 10 03:30:19 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 10 Dec 2013 12:30:19 +1000 Subject: [Distutils] Setuptools 2.0 Released In-Reply-To: References: <8bdbbc71abf04a2f9fd54223e1dd46c7@BLUPR06MB434.namprd06.prod.outlook.com> Message-ID: On 10 Dec 2013 08:03, "Tres Seaver" wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/09/2013 04:40 PM, Antoine Pitrou wrote: > > Is there a changelog? > > https://bitbucket.org/pypa/setuptools/src/f4d6757ae9fa79df03ef6ed070ab8757707fe963/CHANGES.txt?at=default Ah, one instance of the legacy import system PEP 302 emulation finally removed :) Perhaps for 3.5, I (or someone else!) will finally get around to making the emulation layer in pkgutil a thin wrapper around importlib. Since the import system migrated to importlib in 3.3, the utility functions in pkgutil no longer rely on that emulation, but it's still around for backwards compatibility with direct third party usage. Cheers, Nick. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlKmPi0ACgkQ+gerLs4ltQ6HggCguPzkufm/VmuORsxC5xJ59rwQ > yY4An2y622rRFmNFrNOxEit1TED0QpMw > =P0Ix > -----END PGP SIGNATURE----- > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Wed Dec 11 12:29:05 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 11 Dec 2013 11:29:05 +0000 Subject: [Distutils] problems upload docs to pythonhosted.org with setuptools 2.0 Message-ID: <52A84C81.7090902@simplistix.co.uk> Hi All, I'm attempting to upload docs for a new release of my package to pythonhosted.org but I get: $ bin/docpy setup.py upload_docs --upload-dir=docs/_build/html running upload_docs Submitting documentation to http://pypi.python.org/pypi Upload failed (400): Form data is not correctly encoded in UTF-8 My docs are all ascii as far as I know. I'm using setuptools 2.0 on Python 2.7.3. Downgrading to setuptools 1.4.2 fixed the problem. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From bkabrda at redhat.com Thu Dec 12 11:43:01 2013 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 12 Dec 2013 05:43:01 -0500 (EST) Subject: [Distutils] Scripts in installed wheel RECORD In-Reply-To: <405473225.15595232.1386844263466.JavaMail.root@redhat.com> Message-ID: <1379220887.15598707.1386844981911.JavaMail.root@redhat.com> Hey all, so I noticed a strange thing in RECORD of an installed wheel: When a wheel that contains file "foo-1.data/scripts/spam" is installed, "spam" gets copied to a proper dir (e.g. /usr/bin) and the whole "foo-1.data" directory is deleted. A line with the new location is added to RECORD, but the old line is not deleted, so the file is listed twice in the RECORD file. This seems to be wrong, since the original file doesn't exist any more, so "foo-1.data/scripts/spam" should be deleted from the RECORD (applies to all all files in "foo-1.data"). PEP 427 doesn't mention this situation, so I'm not sure - is this expected? Thanks. -- Regards, Bohuslav "Slavek" Kabrda. From p.f.moore at gmail.com Thu Dec 12 12:36:09 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 12 Dec 2013 11:36:09 +0000 Subject: [Distutils] Scripts in installed wheel RECORD In-Reply-To: <1379220887.15598707.1386844981911.JavaMail.root@redhat.com> References: <405473225.15595232.1386844263466.JavaMail.root@redhat.com> <1379220887.15598707.1386844981911.JavaMail.root@redhat.com> Message-ID: On 12 December 2013 10:43, Bohuslav Kabrda wrote: > Hey all, > so I noticed a strange thing in RECORD of an installed wheel: > When a wheel that contains file "foo-1.data/scripts/spam" is installed, "spam" gets copied to a proper dir (e.g. /usr/bin) and the whole "foo-1.data" directory is deleted. A line with the new location is added to RECORD, but the old line is not deleted, so the file is listed twice in the RECORD file. > This seems to be wrong, since the original file doesn't exist any more, so "foo-1.data/scripts/spam" should be deleted from the RECORD (applies to all all files in "foo-1.data"). > PEP 427 doesn't mention this situation, so I'm not sure - is this expected? That doesn't sound like expected behaviour. I'd suggest you raise a bug against whichever toold you used to install the wheel (pip, probably, but maybe wheel or distlib I guess) with the steps needed to demonstrate the issue. Just checking my installation, it looks like it may be pip doing this - I see one RECORD file that looks similar in my site-packages. Please do raise an issue on the pip tracker, so it doesn't get overlooked. Paul From bkabrda at redhat.com Thu Dec 12 13:13:04 2013 From: bkabrda at redhat.com (Bohuslav Kabrda) Date: Thu, 12 Dec 2013 07:13:04 -0500 (EST) Subject: [Distutils] Scripts in installed wheel RECORD In-Reply-To: References: <405473225.15595232.1386844263466.JavaMail.root@redhat.com> <1379220887.15598707.1386844981911.JavaMail.root@redhat.com> Message-ID: <1649839206.15686964.1386850384279.JavaMail.root@redhat.com> ----- Original Message ----- > On 12 December 2013 10:43, Bohuslav Kabrda wrote: > > Hey all, > > so I noticed a strange thing in RECORD of an installed wheel: > > When a wheel that contains file "foo-1.data/scripts/spam" is installed, > > "spam" gets copied to a proper dir (e.g. /usr/bin) and the whole > > "foo-1.data" directory is deleted. A line with the new location is added > > to RECORD, but the old line is not deleted, so the file is listed twice in > > the RECORD file. > > This seems to be wrong, since the original file doesn't exist any more, so > > "foo-1.data/scripts/spam" should be deleted from the RECORD (applies to > > all all files in "foo-1.data"). > > PEP 427 doesn't mention this situation, so I'm not sure - is this expected? > > That doesn't sound like expected behaviour. I'd suggest you raise a > bug against whichever toold you used to install the wheel (pip, > probably, but maybe wheel or distlib I guess) with the steps needed to > demonstrate the issue. > > Just checking my installation, it looks like it may be pip doing this > - I see one RECORD file that looks similar in my site-packages. Please > do raise an issue on the pip tracker, so it doesn't get overlooked. Done: https://github.com/pypa/pip/issues/1376 > Paul -- Regards, Bohuslav "Slavek" Kabrda. From lele at metapensiero.it Thu Dec 12 21:11:36 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Thu, 12 Dec 2013 21:11:36 +0100 Subject: [Distutils] Encoding error installing my own package from PyPI Message-ID: <877gb9ommf.fsf@nautilus.nautilus> Hi all, for the first time I'm trying to use PyPI to distribute one of my packages, and I'm having encoding problems with the package metadata when I install it in a Python 3.3 environment using pip 1.4.1. I tried to google around, but didn't find any definitive solution. The package setup.py[1] builds the long_description reading and concatenating the content of two files, respectively README.rst and CHANGES.rst, both encoded in UTF-8. A first attempt used a plain open() call to read the files failed and I thought to understand the reason, as CHANGES.rst effectively contains a few non-ascii characters. I then used codecs.open() with an explicit utf-8 encoding, but that too fails, although in a different place: Downloading/unpacking metapensiero.sqlalchemy.proxy==1.9.6 (from -r requirements.txt (line 15)) Downloading metapensiero.sqlalchemy.proxy-1.9.6.tar.gz Running setup.py egg_info for package metapensiero.sqlalchemy.proxy Cleaning up... Exception: Traceback (most recent call last): File "/opt/python333/lib/python3.3/site-packages/pip/basecommand.py", line 134, in main status = self.run(options, args) File "/opt/python333/lib/python3.3/site-packages/pip/commands/install.py", line 236, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/opt/python333/lib/python3.3/site-packages/pip/req.py", line 1139, in prepare_files req_to_install.assert_source_matches_version() File "/opt/python333/lib/python3.3/site-packages/pip/req.py", line 394, in assert_source_matches_version version = self.installed_version File "/opt/python333/lib/python3.3/site-packages/pip/req.py", line 390, in installed_version return self.pkg_info()['version'] File "/opt/python333/lib/python3.3/site-packages/pip/req.py", line 357, in pkg_info data = self.egg_info_data('PKG-INFO') File "/opt/python333/lib/python3.3/site-packages/pip/req.py", line 297, in egg_info_data data = fp.read() File "/opt/python333/lib/python3.3/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3072: ordinal not in range(128) and this time I cannot see a way out, as egg_info_data() does fp = open(filename, 'r') data = fp.read() to read PKG-INFO content, that AFAICT was encoded in utf-8 by the sdist command... So the question is: how can I solve the issue? Should I make sure that all the meta data information is plain ASCII (which of course I can do quite easily), or am I doing something wrong? Thanks in advance for any hint, ciao, lele. [1] https://bitbucket.org/lele/metapensiero.sqlalchemy.proxy/src/608bf172973986d003e953ba4e46448363f60e44/setup.py?at=master -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From merwok at netwok.org Fri Dec 13 00:02:19 2013 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 12 Dec 2013 18:02:19 -0500 Subject: [Distutils] Encoding error installing my own package from PyPI In-Reply-To: <877gb9ommf.fsf@nautilus.nautilus> References: <877gb9ommf.fsf@nautilus.nautilus> Message-ID: <52AA407B.8040208@netwok.org> Distutils is not fully aware of unicode. Something to try: open your files like you do and get their contents as unicode, but encode the result to UTF-8 bytes before passing it to the setup function. I thinks this is worth reporting to bugs.python.org, if it isn?t already. Regards From dholth at gmail.com Fri Dec 13 14:20:05 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 13 Dec 2013 08:20:05 -0500 Subject: [Distutils] Encoding error installing my own package from PyPI In-Reply-To: <52AA407B.8040208@netwok.org> References: <877gb9ommf.fsf@nautilus.nautilus> <52AA407B.8040208@netwok.org> Message-ID: This is a pip bug. Could Python 3.3 be doing something different or is your default encoding for open() different than the typical pip developer's? https://github.com/pypa/pip/issues/1298 On Thu, Dec 12, 2013 at 6:02 PM, ?ric Araujo wrote: > Distutils is not fully aware of unicode. Something to try: open your files > like you do and get their contents as unicode, but encode the result to > UTF-8 bytes before passing it to the setup function. > > I thinks this is worth reporting to bugs.python.org, if it isn?t already. > > Regards > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From lele at metapensiero.it Fri Dec 13 18:40:26 2013 From: lele at metapensiero.it (Lele Gaifax) Date: Fri, 13 Dec 2013 18:40:26 +0100 Subject: [Distutils] Encoding error installing my own package from PyPI References: <877gb9ommf.fsf@nautilus.nautilus> <52AA407B.8040208@netwok.org> Message-ID: <87wqj8tzsl.fsf@nautilus.nautilus> Daniel Holth writes: > On Thu, Dec 12, 2013 at 6:02 PM, ?ric Araujo wrote: >> Distutils is not fully aware of unicode. Something to try: open your files >> like you do and get their contents as unicode, but encode the result to >> UTF-8 bytes before passing it to the setup function. >> >> I thinks this is worth reporting to bugs.python.org, if it isn?t already. > This is a pip bug. Could Python 3.3 be doing something different or is > your default encoding for open() different than the typical pip > developer's? > > https://github.com/pypa/pip/issues/1298 FTR, no, ?ric suggestion didn't work, it just failed very earlier in the process, at sdist time. After a few other iterations, I was able to find the culprit: as said, I was trying to install it into a fresh Python 3.3.3 environment, within a virtual machine which system locales weren't properly configured for some reason; after a "dpkg-reconfigure locales" and setting the LANG environment variable, everything went smooth! I don't know if #1298 is related to the same problem (will report my experience there, anyway). Thanks a lot, ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From dholth at gmail.com Fri Dec 13 18:45:42 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 13 Dec 2013 12:45:42 -0500 Subject: [Distutils] Encoding error installing my own package from PyPI In-Reply-To: <87wqj8tzsl.fsf@nautilus.nautilus> References: <877gb9ommf.fsf@nautilus.nautilus> <52AA407B.8040208@netwok.org> <87wqj8tzsl.fsf@nautilus.nautilus> Message-ID: On Fri, Dec 13, 2013 at 12:40 PM, Lele Gaifax wrote: > Daniel Holth writes: > >> On Thu, Dec 12, 2013 at 6:02 PM, ?ric Araujo wrote: >>> Distutils is not fully aware of unicode. Something to try: open your files >>> like you do and get their contents as unicode, but encode the result to >>> UTF-8 bytes before passing it to the setup function. >>> >>> I thinks this is worth reporting to bugs.python.org, if it isn?t already. > >> This is a pip bug. Could Python 3.3 be doing something different or is >> your default encoding for open() different than the typical pip >> developer's? >> >> https://github.com/pypa/pip/issues/1298 > > FTR, no, ?ric suggestion didn't work, it just failed very earlier in the > process, at sdist time. > > After a few other iterations, I was able to find the culprit: as said, I was > trying to install it into a fresh Python 3.3.3 environment, within a > virtual machine which system locales weren't properly configured for > some reason; after a "dpkg-reconfigure locales" and setting the LANG > environment variable, everything went smooth! > > I don't know if #1298 is related to the same problem (will report my > experience there, anyway). > > Thanks a lot, > ciao, lele. It is still a pip bug. Portable Python programs should never open files without an explicit encoding because doing so creates bugs that you only find when you ssh into a machine configured for the German language from your English-language machine or whatever. Glad you were able to resolve. From chris at simplistix.co.uk Fri Dec 13 19:01:29 2013 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 13 Dec 2013 18:01:29 +0000 Subject: [Distutils] problems upload docs to pythonhosted.org with setuptools 2.0 In-Reply-To: <52A84C81.7090902@simplistix.co.uk> References: <52A84C81.7090902@simplistix.co.uk> Message-ID: <52AB4B79.4000603@simplistix.co.uk> Hmm, doesn't appear this got any response. Where should I report it? Chris On 11/12/2013 11:29, Chris Withers wrote: > Hi All, > > I'm attempting to upload docs for a new release of my package to > pythonhosted.org but I get: > > $ bin/docpy setup.py upload_docs --upload-dir=docs/_build/html > running upload_docs > Submitting documentation to http://pypi.python.org/pypi > Upload failed (400): Form data is not correctly encoded in UTF-8 > > My docs are all ascii as far as I know. > > I'm using setuptools 2.0 on Python 2.7.3. > > Downgrading to setuptools 1.4.2 fixed the problem. > > Chris > -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ncoghlan at gmail.com Sat Dec 14 06:53:38 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 14 Dec 2013 15:53:38 +1000 Subject: [Distutils] problems upload docs to pythonhosted.org with setuptools 2.0 In-Reply-To: <52AB4B79.4000603@simplistix.co.uk> References: <52A84C81.7090902@simplistix.co.uk> <52AB4B79.4000603@simplistix.co.uk> Message-ID: On 14 December 2013 04:01, Chris Withers wrote: > Hmm, doesn't appear this got any response. > > Where should I report it? The setuptools issue tracker is at https://bitbucket.org/pypa/setuptools/ Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Sun Dec 15 12:20:18 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 15 Dec 2013 11:20:18 +0000 (GMT) Subject: [Distutils] distlib 0.1.5 released on PyPI Message-ID: <1387106418.22897.YahooMailBasic@web171403.mail.ir2.yahoo.com> I've just released version 0.1.5 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Used dummy_threading when threading isn?t available. Used uncompressed Windows executable launchers, because the compressed ones cause false positive warnings with some anti-virus products. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.5 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new From philippe.muller at gmail.com Mon Dec 16 13:28:28 2013 From: philippe.muller at gmail.com (Philippe Muller) Date: Mon, 16 Dec 2013 13:28:28 +0100 Subject: [Distutils] Source distribution fails when packages have a setup.cfg file and are checked out from Perforce Message-ID: Hello, My code is versionned using Perforce, and when creating a source distribution, I get this error : $ python setup.py sdist [...] copying setup.cfg -> foo-0.4.0 Writing foo-0.4.0/setup.cfg error: foo-0.4.0/setup.cfg: Permission denied That's happening because Perforce forces files to be read-only when not "Perforce opened". If I do a "p4 open setup.cfg" before creating the source distribution, the setup.cfg file will be writable, so will its copy, and the source distribution is created without any issue. I currently use a more hackish workaround : chmod +w setup.cfg ; python setup.py sdist ; chmod -w setup.cfg But that's ugly. Does anyone know a cleaner way to bypass this issue ? Thanks, Philippe Muller -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 21 11:57:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Dec 2013 20:57:32 +1000 Subject: [Distutils] Platform specific destinations in wheel files? Message-ID: With the Python 3.4 feature freeze behind us, I've started looking at doing a new update of the draft metadata 2.0 docs. Vaguely related to that are the recent discussions about being able to publish an FHS compliant daemon like cobblerd as a wheel file - using Python specific formats to define the layout of full applications, not just libraries. Currently, all of the paths defined in sysconfig schemes aim to be platform neutral. However, this means that some platform specific concepts can't easily be handled by the wheel format - the "data" scheme can sort of handle it, but it isn't particularly intuitive, and doesn't inherently reflect the platform dependent nature of the layout being defined. I'd generally been resisting the idea of supporting this (since I favour interoperating with system packaging tools where appropriate over attempting to replace them entirely), but in this case I'm starting to think it may be necessary to support these layouts in the next version of the wheel format in order to *also* support automated conversion of upstream projects to policy compliant system packages. As a reminder, here are the currently defined Python specific directories relevant to wheel files: platlib: directory for site-specific, platform-specific files. purelib: directory for site-specific, non-platform-specific files. include: directory for non-platform-specific header files. platinclude: directory for platform-specific header files. And these are the generic directories that aren't necessarily Python specific: scripts: directory for script files. data: directory for data files. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the ".data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new ".app" subdirectory in parallel. A wheel that had content in "app" would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. All POSIX systems would at least support the "fhs" subdirectory. For a system install, this would map to "/", for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to "~/.local". I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Dec 21 12:40:19 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 21 Dec 2013 11:40:19 +0000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 21 December 2013 10:57, Nick Coghlan wrote: > And these are the generic directories that aren't necessarily Python specific: > > scripts: directory for script files. > data: directory for data files. It's worth noting that "data", although in essence a platform neutral term, is in practice not very well defined. Projects as a result use it inconsistently, and usually based on their expectations from the platforms they are used to. One thing I have seen data being used for a lot is documentation - and not consistently - some projects use /man, others use /share/man, and others /. I'd suggest the following changes in this area: 1. Add a "docs" directory to hold documentation. We may need to have a debate on how this should be structured, and how to handle backward compatibility, but this is a genuine need that the current scheme does not cover. 2. Add some more explicit wording on how "data" should (and should not) be used. For instance, having FHS-style structures like share/man/man1 under "data" doesn't really help with cross-platform compatibility that much - it just provides somewhere to dump the data on Windows while still interoperating with standard tools on a POSIX system. > This is still only a half-baked idea at this point, but I'm currently > leaning towards keeping the ".data" sysconfig subdirectories in > the wheel format cross platform (and somewhat Python specific), and > adding a new ".app" subdirectory in parallel. > > A wheel that had content in "app" would be inherently platform > specific - you wouldn't be permitted to use the new directory in a > cross-platform wheel file. The defined subdirectories of app would > also be platform specific. It might be worth having the possibility of optional parts of "app". To use the docs example again, it would be easy to see projects putting Unix manpages in app, and so unintentinally making the distribution platform specific, when in practice it would be perfectly OK to simply omit the manpages on Windows. Other app files (for example config files in etc) *will* be essential for proper operation, of course. > All POSIX systems would at least support the "fhs" subdirectory. For a > system install, this would map to "/", for a virtualenv install it > would map to the root of the virtualenv and for a user install it > would map to "~/.local". > > I'm not sure what other subdirectories would be appropriate for > Windows or Mac OS X, although I expect being able to install into > Program Files and Application Data would be interesting for Windows > apps, and into an application folder for Mac OS X. I'm not sure Program Files is that useful - very few apps would install into *both* Program Files and an application area (site-packages). But Application Data might be useful. Another possibility would be the start menu - but that's not precisely a filesystem location, you'd need to put metadata into the "Start Menu" area of the wheel, and use the Windows API to create the start menu items at install time. > It's really the potential for FHS support that drives my interest in > the idea, but if we're going to do something like this at all, it > shouldn't be POSIX specific. Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under "app" is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting. Maybe this is something that should be handled by the post-install hook support. The "app" area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather: 1. Tools must make the "app" area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a temporary location on disk, and tell the hooks the location of that location) 2. Tools should provide an extensibility mechanism for installing system-wide processors for app data (so that we don't need every wheel to ship the FHS-handling post-install hooks) 3. Standard processing for particular app areas (for example, app/fhs) could be defined independently of the wheel spec. One of the motivations I have here is to keep the install behaviour from getting too complex. We need to remember that pip is not the only wheel installer (there's distlib and wheel itself). So I don't want to mandate too much complexity into the install phase (for example, script wrappers aren't handled in wheel install, and I'm not sure what distlib does - so there's implementation differences creeping in right now). Paul From ncoghlan at gmail.com Sat Dec 21 13:01:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 21 Dec 2013 22:01:11 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 21 December 2013 21:40, Paul Moore wrote: > On 21 December 2013 10:57, Nick Coghlan wrote: >> And these are the generic directories that aren't necessarily Python specific: >> >> scripts: directory for script files. >> data: directory for data files. > > It's worth noting that "data", although in essence a platform neutral > term, is in practice not very well defined. Projects as a result use > it inconsistently, and usually based on their expectations from the > platforms they are used to. One thing I have seen data being used for > a lot is documentation - and not consistently - some projects use > /man, others use /share/man, and others > /. > > I'd suggest the following changes in this area: > > 1. Add a "docs" directory to hold documentation. We may need to have a > debate on how this should be structured, and how to handle backward > compatibility, but this is a genuine need that the current scheme does > not cover. Evolution of sysconfig as we go forward is indeed something for us to think about. It may be a case where we need to actually decouple the wheel spec from the implicit dependency on those definitions. The stdlib related paths don't make sense to use in a wheel file, so what probably makes sense is for us to make just purelib, platlib, include and platinclude depend on sysconfig (since they're installing to Python-specific locations and need the assistance from the standard library), while other paths like scripts and docs would be defined independently of sysconfig. > 2. Add some more explicit wording on how "data" should (and should > not) be used. For instance, having FHS-style structures like > share/man/man1 under "data" doesn't really help with cross-platform > compatibility that much - it just provides somewhere to dump the data > on Windows while still interoperating with standard tools on a POSIX > system. I'd be interesting in reading such guidance, because I really have no idea... (I'd never even noticed the option until Daniel pointed it out to me recently) >> This is still only a half-baked idea at this point, but I'm currently >> leaning towards keeping the ".data" sysconfig subdirectories in >> the wheel format cross platform (and somewhat Python specific), and >> adding a new ".app" subdirectory in parallel. >> >> A wheel that had content in "app" would be inherently platform >> specific - you wouldn't be permitted to use the new directory in a >> cross-platform wheel file. The defined subdirectories of app would >> also be platform specific. > > It might be worth having the possibility of optional parts of "app". > To use the docs example again, it would be easy to see projects > putting Unix manpages in app, and so unintentinally making the > distribution platform specific, when in practice it would be perfectly > OK to simply omit the manpages on Windows. Other app files (for > example config files in etc) *will* be essential for proper operation, > of course. I think a separate docs path would be a better solution for that. There may be a couple of others we decide to support directly (like an admin_scripts path that maps to sbin under FHS and to the same place that scripts maps to on Windows). >> All POSIX systems would at least support the "fhs" subdirectory. For a >> system install, this would map to "/", for a virtualenv install it >> would map to the root of the virtualenv and for a user install it >> would map to "~/.local". >> >> I'm not sure what other subdirectories would be appropriate for >> Windows or Mac OS X, although I expect being able to install into >> Program Files and Application Data would be interesting for Windows >> apps, and into an application folder for Mac OS X. > > I'm not sure Program Files is that useful - very few apps would > install into *both* Program Files and an application area > (site-packages). But Application Data might be useful. Another > possibility would be the start menu - but that's not precisely a > filesystem location, you'd need to put metadata into the "Start Menu" > area of the wheel, and use the Windows API to create the start menu > items at install time. > >> It's really the potential for FHS support that drives my interest in >> the idea, but if we're going to do something like this at all, it >> shouldn't be POSIX specific. > > Agreed in principle. But in practice, do you expect the wheel spec to > define exactly how each scheme under "app" is mapped? And do you > expect every tool to implement those features? That implies a lot of > work getting consensus and a change to the wheel spec every time > someone thinks of a new platform-specific area that's worth > supporting. > > Maybe this is something that should be handled by the post-install > hook support. The "app" area could be an area that is reserved in the > wheel for platform-specific data as you say, but there is no specified > behaviour for tools to implement. Rather: > > 1. Tools must make the "app" area accessible in post-install hooks > (how is yet to be defined, simplest answer just put it in a temporary > location on disk, and tell the hooks the location of that location) > 2. Tools should provide an extensibility mechanism for installing > system-wide processors for app data (so that we don't need every wheel > to ship the FHS-handling post-install hooks) > 3. Standard processing for particular app areas (for example, app/fhs) > could be defined independently of the wheel spec. > > One of the motivations I have here is to keep the install behaviour > from getting too complex. We need to remember that pip is not the only > wheel installer (there's distlib and wheel itself). So I don't want to > mandate too much complexity into the install phase (for example, > script wrappers aren't handled in wheel install, and I'm not sure what > distlib does - so there's implementation differences creeping in right > now). Oh, I like that idea. Let me get the next draft of the metadata 2.0 PEPs cleaned up and published, and then we can discuss it further (I think it ties into nicely with some of the changes I've made in response to PJE's feedback on the last draft). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Dec 21 15:46:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Dec 2013 00:46:37 +1000 Subject: [Distutils] Major update to the metadata 2.0 draft PEPs Message-ID: I've just published the first draft of the metadata 2.0 spec that moves all of the fields that aren't part of the core metadata or potentially needed for dependency resolution out to a separate "standard metadata extensions" PEP. I think this makes PEP 426 itself substantially less overwhelming, and also provides convenient names to refer to the various other subsets of the metadata. The new PEP defining the standard extensions is PEP 459. Metadata 2.0: http://www.python.org/dev/peps/pep-0426/ Versioning: http://www.python.org/dev/peps/pep-0440/ Standard Extensions: http://www.python.org/dev/peps/pep-0459/ You can see the full deltas for PEP 426 and PEP 440 here: http://hg.python.org/peps/rev/d18629859154 Or slightly more incremental deltas in the BitBucket repo: https://bitbucket.org/pypa/pypi-metadata-formats/overview With this update, I believe PEP 440 is extremely close to acceptance, as I have tried to resolve a number of problems Donald noted while working on Warehouse, as well as the perennial problem of how to deal with versioning of packages where Linux distro vendors (or other system integrators) have applied additional patches. The PEP 440 changes: - added the "integrator suffix" (essentially an extra release segment, separated from the upstream version number by "-"). It is effectively ignored by most of the matching operators, but sorts as a higher version than the version with no suffix. - allowed date based versions for most purposes, but still reject them for the "~=" compatible version operator. The implied comparison operator for date based versions is ">=". - clarified that zeroes and leading zeroes are permitted and that they should be sorted like numbers - based on discussions with some of the Red Hat security folks, I tweaked the direct URL references to *always* require a hash in the URL, even when using HTTPS or SSH. PEP 426 and PEP 459 are further away from finalisation, but we're definitely getting closer to something I'm happy with. The PEP 426 changes: - many of the fields are gone, having moved to PEP 459 extensions - this also allowed the notion of "essential dependency metadata" (with it's own separate filename) to go away - the extension format changed to require a submapping - extensions are now versioned (with an implied 1.0 if not specified) - as with direct URLs in PEP 440, source URLs are now required to contain a hash PEP 459 overview: - standard extensions using the "python.*" namespace - the details, project, exports and commands extensions just lift several fields out of PEP 426 - the integrator extension is new, and matches the project extension structure. It allows redistributors like a Linux distro or a Python vendor to add their own info without overwriting the upstream project metadata - the new metadata_hooks extension replaces the install hooks design in earlier drafts of PEP 426. In this new scheme, Twisted, for example, would define an appropriate export group hook in order to be told about any new extensions that exported Twisted plugins and take appropriate action In this revised metadata model, distributions *will* trigger their own postinstall metadata hooks, but won't trigger their postuninstall hooks. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Dec 21 17:08:08 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 22 Dec 2013 02:08:08 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 21 December 2013 22:01, Nick Coghlan wrote: > On 21 December 2013 21:40, Paul Moore wrote: >> On 21 December 2013 10:57, Nick Coghlan wrote: >>> It's really the potential for FHS support that drives my interest in >>> the idea, but if we're going to do something like this at all, it >>> shouldn't be POSIX specific. >> >> Agreed in principle. But in practice, do you expect the wheel spec to >> define exactly how each scheme under "app" is mapped? And do you >> expect every tool to implement those features? That implies a lot of >> work getting consensus and a change to the wheel spec every time >> someone thinks of a new platform-specific area that's worth >> supporting. >> >> Maybe this is something that should be handled by the post-install >> hook support. The "app" area could be an area that is reserved in the >> wheel for platform-specific data as you say, but there is no specified >> behaviour for tools to implement. Rather: >> >> 1. Tools must make the "app" area accessible in post-install hooks >> (how is yet to be defined, simplest answer just put it in a temporary >> location on disk, and tell the hooks the location of that location) >> 2. Tools should provide an extensibility mechanism for installing >> system-wide processors for app data (so that we don't need every wheel >> to ship the FHS-handling post-install hooks) >> 3. Standard processing for particular app areas (for example, app/fhs) >> could be defined independently of the wheel spec. >> >> One of the motivations I have here is to keep the install behaviour >> from getting too complex. We need to remember that pip is not the only >> wheel installer (there's distlib and wheel itself). So I don't want to >> mandate too much complexity into the install phase (for example, >> script wrappers aren't handled in wheel install, and I'm not sure what >> distlib does - so there's implementation differences creeping in right >> now). > > Oh, I like that idea. Let me get the next draft of the metadata 2.0 > PEPs cleaned up and published, and then we can discuss it further (I > think it ties into nicely with some of the changes I've made in > response to PJE's feedback on the last draft). The revised proposal for the hooks mechanism is now up as part of the new standard metadata extensions PEP: http://www.python.org/dev/peps/pep-0459/#the-metadata-hooks-extension This currently just covers step 2 - there's no mechanism for giving such hooks access to files without first putting them in one of the standard locations. It's the first draft of that design though, so I expect it will need a few iterations before we're happy with it. For example, I think a slight tweak to the metadata_hooks extension could extend it to cover this case as well: support a "custom_path" option as a new kind of hook definition. So, for example, you would be able to trap the fhs path this way: "python.metadata_hooks": { "custom_paths": { "fhs": { "postinstall": "fedora_pip.plugins:install_fhs_files", } } } In this case, the signature of the postinstall hook would be slightly different from the one defined in the PEP for extensions and export_groups, since it would also need to include the path to a temporary directory containing the extracted files. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sun Dec 22 17:54:30 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Dec 2013 16:54:30 +0000 (UTC) Subject: [Distutils] classifiers not up to date on testpypi Message-ID: Registering pathlib to https://testpypi.python.org/pypi Server response (400): Invalid classifier "Programming Language :: Python :: 3.4" From chris.barker at noaa.gov Mon Dec 23 21:53:52 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 23 Dec 2013 12:53:52 -0800 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On Sat, Dec 21, 2013 at 2:57 AM, Nick Coghlan wrote: > compliant daemon like cobblerd as a wheel file - using Python specific > formats to define the layout of full applications, not just libraries. > > I'd generally been resisting the idea of supporting this (since I > favour interoperating with system packaging tools where appropriate > over attempting to replace them entirely), but in this case I'm > starting to think it may be necessary to support these layouts in the > next version of the wheel format in order to *also* support automated > conversion of upstream projects to policy compliant system packages. > > hmm I tend to think, like you, that this isn't a problem wheel should solve, but can also see the advantages...for the moment though talking about how it would solve it may help clarify whether it should. adding a new ".app" subdirectory in parallel. > > A wheel that had content in "app" would be inherently platform > specific - you wouldn't be permitted to use the new directory in a > cross-platform wheel file. The defined subdirectories of app would > also be platform specific. > is this necessary -- couldn't there be a way to provide the info in a cross-platform way, and have it mapped to the platform specific locations at install-time? > All POSIX systems would at least support the "fhs" subdirectory. For a > system install, this would map to "/", for a virtualenv install it > would map to the root of the virtualenv and for a user install it > would map to "~/.local". > > then you explicitly put in "bin", sbin", "share", whatever? This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things.... Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed: executables (bin) system_executables (sbin) user_executables (./bin) documentation (doc) system_data_files (share ?) user_data_files (./share ) app_configuration (/etc/appname) user_app_configuration (./.app_name : ./Library ) .... This could end up being a pretty big list, but I think it could be finite. Then at install-time, the installer maps these to the appropriate place on the system. It's a little different application, but wxWidgets does this pretty successfully with wxStandardPaths. -Chris > I'm not sure what other subdirectories would be appropriate for > Windows or Mac OS X, although I expect being able to install into > Program Files and Application Data would be interesting for Windows > apps, and into an application folder for Mac OS X. > > It's really the potential for FHS support that drives my interest in > the idea, but if we're going to do something like this at all, it > shouldn't be POSIX specific. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Dec 23 22:22:14 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 23 Dec 2013 21:22:14 +0000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 23 December 2013 20:53, Chris Barker wrote: > then you explicitly put in "bin", sbin", "share", whatever? > > This seems really klunky to me, and also forces platform dependence, and is > fundamentally tied to how posix does things.... > > Maybe it's not possible, but I suggest that we could pre-define the > locations that might be needed: Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. This has been a very common issue historically, and I'm very reluctant to add anything that makes it easier to limit usability of packages across platforms. Portable by default should be the principle. But I see there is a need, and I understand that practicality beats purity. Let's just make sure that it's more effort to use the platform-specific approach than not to, so it doesn't become an attractive nuisance. BTW, I like your idea of defining generic locations - but in the past efforts to do this have been doomed to death by bikeshedding and platform wars. So while I agree in principle, I fear it may not happen in practice :-( Paul From dholth at gmail.com Mon Dec 23 22:43:16 2013 From: dholth at gmail.com (Daniel Holth) Date: Mon, 23 Dec 2013 16:43:16 -0500 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On Mon, Dec 23, 2013 at 4:22 PM, Paul Moore wrote: > On 23 December 2013 20:53, Chris Barker wrote: >> then you explicitly put in "bin", sbin", "share", whatever? >> >> This seems really klunky to me, and also forces platform dependence, and is >> fundamentally tied to how posix does things.... >> >> Maybe it's not possible, but I suggest that we could pre-define the >> locations that might be needed: > > Agreed. My biggest concern with this whole idea is that developers > (typically POSIX developers, but it applies equally to all) will > *think* they need something like sbin because they are used to the > concept from their environment, and so write their wheel to use it and > hence be platform specific. However, with a little thought (possibly > hardly any thought in the case of sbin :-)) they could have chosen a > more generic approach which makes their project available to users of > other platforms. > > This has been a very common issue historically, and I'm very reluctant > to add anything that makes it easier to limit usability of packages > across platforms. Portable by default should be the principle. > > But I see there is a need, and I understand that practicality beats > purity. Let's just make sure that it's more effort to use the > platform-specific approach than not to, so it doesn't become an > attractive nuisance. "Attractive nuisance" is the first phrase that comes to mind when I think of the FHS. > BTW, I like your idea of defining generic locations - but in the past > efforts to do this have been doomed to death by bikeshedding and > platform wars. So while I agree in principle, I fear it may not happen > in practice :-( > > Paul The GNU autoconf paths are the obvious choice. It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html From chris.barker at noaa.gov Tue Dec 24 18:50:59 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 24 Dec 2013 09:50:59 -0800 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth > Agreed. My biggest concern with this whole idea is that developers > > (typically POSIX developers, but it applies equally to all) will > > *think* they need something like sbin because they are used to the > > concept from their environment, and so write their wheel to use it and > > hence be platform specific. > exactly. However, with a little thought (possibly > > hardly any thought in the case of sbin :-)) they could have chosen a > > more generic approach which makes their project available to users of > > other platforms. > right, but setting the system up to allow a prefix and then hard-specify paths under that makes it impossible to do it cross-platform! > Portable by default should be the principle. > +1 ... and at least possible (without writing multiple platform-specific versions...)! ANother thought is: who should control where things are put? a) the package developer? b) the python system maintainer? I think clearly the answer is (b). i.e. we'll have reasonable defaults in the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro..... could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that. The GNU autoconf paths are the obvious choice. Certainly a good palce to start. One issue -- that (and the FHS) is built around the idea of a "prefix", where you can shift the whole pile to /usr or /usr/local, or /opt, or??? But that concept doesn't work on all platforms, so we should be careful about isolating it. -Chris > It would be really easy > to try adding these to the dict that wheel installers use to relocate > stuff. > http://www.gnu.org/prep/standards/html_node/Directory-Variables.html > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 24 23:28:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 25 Dec 2013 08:28:27 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 25 Dec 2013 04:14, "Chris Barker" wrote: > > On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth > Agreed. My biggest concern with this whole idea is that developers >> >> > (typically POSIX developers, but it applies equally to all) will >> > *think* they need something like sbin because they are used to the >> > concept from their environment, and so write their wheel to use it and >> > hence be platform specific. > > > exactly. > >> However, with a little thought (possibly >> > hardly any thought in the case of sbin :-)) they could have chosen a >> > more generic approach which makes their project available to users of >> > other platforms. > > > right, but setting the system up to allow a prefix and then hard-specify paths under that makes it impossible to do it cross-platform! > >> > Portable by default should be the principle. > > > +1 ... and at least possible (without writing multiple platform-specific versions...)! > > ANother thought is: > > who should control where things are put? > > a) the package developer? > b) the python system maintainer? > > I think clearly the answer is (b). i.e. we'll have reasonable defaults in the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro..... could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that. > >> The GNU autoconf paths are the obvious choice. > > > Certainly a good palce to start. > > One issue -- that (and the FHS) is built around the idea of a "prefix", where you can shift the whole pile to /usr or /usr/local, or /opt, or??? > > But that concept doesn't work on all platforms, so we should be careful about isolating it. Encapsulating that assumption is why I think the "gnu" nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows. Cheers, Nick. > > -Chris > > >> >> It would be really easy >> to try adding these to the dict that wheel installers use to relocate >> stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html > > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Fri Dec 27 03:34:44 2013 From: qwcode at gmail.com (Marcus Smith) Date: Thu, 26 Dec 2013 18:34:44 -0800 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: > > > data: directory for data files. > > This is still only a half-baked idea at this point, but I'm currently > leaning towards keeping the ".data" sysconfig subdirectories in > the wheel format cross platform (and somewhat Python specific), and > adding a new ".app" subdirectory in parallel. > It's important to point out that the distutils/setuptools implementation of installing setup "data_files" offers a loophole now to achieve the "app" notion (at least when installing from sdist). When "data_files" paths are absolute, it honors them as absolute, allowing you to install anything/anywhere, if you have privileges (otoh when paths are relative, they go into the sysconfig "data" path as you would expect) as noted, wheel installs have no such mechanism, because there's just "*.data" and it maps to sysconfig "data" in the pip wheel installer. Also note that bdist_wheel actually does the wrong thing now with absolute "data_files" paths by letting them end up relative to site-packages https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files ) but I agree we need something more than "data" for platform-specific data files. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Dec 27 15:04:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 00:04:48 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 27 December 2013 12:34, Marcus Smith wrote: >> >> data: directory for data files. >> >> This is still only a half-baked idea at this point, but I'm currently >> leaning towards keeping the ".data" sysconfig subdirectories in >> the wheel format cross platform (and somewhat Python specific), and >> adding a new ".app" subdirectory in parallel. > > > It's important to point out that the distutils/setuptools implementation of > installing setup "data_files" offers a loophole now to achieve the "app" > notion (at least when installing from sdist). > > When "data_files" paths are absolute, it honors them as absolute, allowing > you to install anything/anywhere, if you have privileges (otoh when paths > are relative, they go into the sysconfig "data" path as you would expect) > > as noted, wheel installs have no such mechanism, because there's just > "*.data" and it maps to sysconfig "data" in the pip wheel installer. Also > note that bdist_wheel actually does the wrong thing now with absolute > "data_files" paths by letting them end up relative to site-packages > https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_files) > > but I agree we need something more than "data" for platform-specific data > files. Yeah, I don't think it's an urgent problem, but I believe it's one we will want to solve eventually. If we do another packaging BoF in Montreal (and hopefully we'll have the numbers for it to make sense to do so), that could be a good opportunity to bounce a few ideas around without spamming the list with more incomplete designs like my initial attempt. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Fri Dec 27 21:02:17 2013 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 27 Dec 2013 12:02:17 -0800 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan wrote: > > But that concept doesn't work on all platforms, so we should be careful > about isolating it. > > Encapsulating that assumption is why I think the "gnu" nesting is > justified. There are layout expectations inherent in the autoconf directory > design that just don't make sense on Windows, so any package expecting them > is going to be a bit quirky if installed on Windows. > I'm confused now as to what has been proposed or being discussed, or... I _thought_ that this thread started with a proposal that package authors would do something like specifying a file hierarchy for the stuff they are delivering: /bin/some_scripts /share/some_data /man/some_docs .... then at installation time, the python distro would decide where to copy all that. But this would beb worthless on non-gnu systems and would require: 1) package writer to write three or four versions 2) wheels to be platform-dependent unnecessarily. So my suggestion was to define the various locations where stuff may need to be installed at a higher level: place_to_put_top_level_scripts place_to_put_documentation place_to_app_data place_to_put_user_configuration_files place_to_put_user_system_configuration_files ..... Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc.... This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ... That may be what you had in mind, but I got confused. -Chris > Cheers, > Nick. > > > > > -Chris > > > > > >> > >> It would be really easy > >> to try adding these to the dict that wheel installers use to relocate > >> stuff. > http://www.gnu.org/prep/standards/html_node/Directory-Variables.html > > > > > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 28 07:00:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 16:00:13 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 28 December 2013 06:02, Chris Barker wrote: > On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan wrote: >> >> > But that concept doesn't work on all platforms, so we should be careful >> > about isolating it. >> >> Encapsulating that assumption is why I think the "gnu" nesting is >> justified. There are layout expectations inherent in the autoconf directory >> design that just don't make sense on Windows, so any package expecting them >> is going to be a bit quirky if installed on Windows. > > I'm confused now as to what has been proposed or being discussed, or... > > I _thought_ that this thread started with a proposal that package authors > would do something like specifying a file hierarchy for the stuff they are > delivering: > > /bin/some_scripts > /share/some_data > /man/some_docs > .... > > then at installation time, the python distro would decide where to copy all > that. > > But this would beb worthless on non-gnu systems and would require: > 1) package writer to write three or four versions > 2) wheels to be platform-dependent unnecessarily. > > So my suggestion was to define the various locations where stuff may need to > be installed at a higher level: > > place_to_put_top_level_scripts > place_to_put_documentation > place_to_app_data > place_to_put_user_configuration_files > place_to_put_user_system_configuration_files > ..... > > Then the python distro would map these to actual paths at install time: gnu > systems would map the gnu locations, Windows to Windows-appropriate > locations, OS-X to OS-X locations, etc.... This wold also allow python > distros like Anaconda or macports python, or ??? to do their own thing, > which may not be putting everything in /usr/local, or ... > > That may be what you had in mind, but I got confused. There's no concrete proposal on the table at this point in time - just an acknowledgement that this is a use case the wheel format doesn't yet handle (sdists can handle it due to the way that setuptools handles the data_files option). However, any concrete proposal will likely use the GNU structure as the basis for "kinds of files" rather than inventing a new Python-specific naming scheme (or make an extremely compelling case for inventing our own custom scheme), and on Windows and Mac OS X, many of those will likely just map to dumping things in the app directory (or the application data directory in the case of Windows). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Dec 28 07:17:27 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 16:17:27 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 28 December 2013 16:00, Nick Coghlan wrote: > On 28 December 2013 06:02, Chris Barker wrote: >> Then the python distro would map these to actual paths at install time: gnu >> systems would map the gnu locations, Windows to Windows-appropriate >> locations, OS-X to OS-X locations, etc.... This wold also allow python >> distros like Anaconda or macports python, or ??? to do their own thing, >> which may not be putting everything in /usr/local, or ... >> >> That may be what you had in mind, but I got confused. > > There's no concrete proposal on the table at this point in time - just > an acknowledgement that this is a use case the wheel format doesn't > yet handle (sdists can handle it due to the way that setuptools > handles the data_files option). > > However, any concrete proposal will likely use the GNU structure as > the basis for "kinds of files" rather than inventing a new > Python-specific naming scheme (or make an extremely compelling case > for inventing our own custom scheme), and on Windows and Mac OS X, > many of those will likely just map to dumping things in the app > directory (or the application data directory in the case of Windows). The other main thing to look at in terms of current state of the art is npm, for both structure and hooks. https://npmjs.org/doc/files/npm-folders.html https://npmjs.org/doc/files/package.json.html https://npmjs.org/doc/misc/npm-scripts.html npm is basically a full-blown platform packaging system (covering config, runtime scripts, etc), but with a heavily POSIX biased bent (even moreso than the Python ecosystem). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sat Dec 28 10:31:56 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 28 Dec 2013 09:31:56 +0000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: References: Message-ID: On 28 December 2013 06:17, Nick Coghlan wrote: > The other main thing to look at in terms of current state of the art > is npm, for both structure and hooks. Another place to look would be existing usage in distributions on PyPI. I know that Vinay has done some work on extracting build-time data from PyPI (the red-dove metadata repo) - I don't know if he has anything like this available. It's probably too much work to get if nobody has already done it as a side effect of something else, though... Paul. From ncoghlan at gmail.com Sat Dec 28 14:18:07 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 28 Dec 2013 23:18:07 +1000 Subject: [Distutils] Platform specific destinations in wheel files? In-Reply-To: <1388230571.79803.YahooMailBasic@web171303.mail.ir2.yahoo.com> References: <1388230571.79803.YahooMailBasic@web171303.mail.ir2.yahoo.com> Message-ID: On 28 December 2013 21:36, Vinay Sajip wrote: > I don't have anything actually written, though all the information is available in the JSON for individual releases under the "source/data-files" key. I will look at writing a simple scanner which looks at all the unique directories declared by PyPI dists. This should be faster when run locally on the server than a script which fetches the JSON via HTTP. Give me a bit of time, and I'll post what I find. Data driven design? What madness is this? :) Good idea Paul, and thanks for looking into it Vinay. Cheers, Nick. P.S. That reminds me, I should redo the PyPI version compatibility numbers for the latest PEP 440 draft - making pytz relatively easy to convert was the main gain, but I suspect the overall compatibility percentage will improve a little bit regardless. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Tue Dec 31 11:37:26 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 31 Dec 2013 10:37:26 +0000 (GMT) Subject: [Distutils] distlib 0.1.6 released on PyPI Message-ID: <1388486246.17012.YahooMailBasic@web171303.mail.ir2.yahoo.com> I've just released version 0.1.6 of distlib on PyPI [1]. For newcomers, distlib is a library of packaging functionality which is intended to be usable as the basis for third-party packaging tools. The main changes in this release are as follows: Updated the binary launcher executables, because the wrong variant of these was shipped with the previous release. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions for improvements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.1.6 [2] http://pythonhosted.org/distlib/overview.html#change-log-for-distlib [3] https://bitbucket.org/pypa/distlib/issues/new From qwcode at gmail.com Tue Dec 31 21:05:53 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 31 Dec 2013 12:05:53 -0800 Subject: [Distutils] migration path for entry_points? Message-ID: ok, so PEP459 has exports: http://www.python.org/dev/peps/pep-0459/#the-exports-extension and distlib seems to be implementing them. and "exports" seem to be entry points 2.0. so theoretically, when PEP426/PEP459 becomes live.... what does that mean for setuptools-based projects? (I understand that we might not have a clear idea now, but just trying to narrow down the possibilities) 1) the setuptools "entry_points" keyword (and pkg_resources api) gets re-implemented to understand PEP459 exports? or 2) "entry_points" keeps the old implementation, and possibly tools are written to handle the old and new metadata? or 3) "entry_points" are stranded, and consumers have to rewrite setup.py files to use some new keyword that understands PEP459 or 4) something altogether different... as for why I'm asking, pip itself is considering command extensions, so it's a direct practical matter for us. Marcus -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Tue Dec 31 21:57:54 2013 From: dholth at gmail.com (Daniel Holth) Date: Tue, 31 Dec 2013 15:57:54 -0500 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: It's nice to have that split up into lots of more focused documents. The comment I have is that I think entry points sometimes have multiple values with the same name (a list or multidict instead of a simple dict) although I'm not sure how common that is... Tools should just do automatic conversion of entry_points.txt to the new metadata format when .dist-info is in use. On Tue, Dec 31, 2013 at 3:05 PM, Marcus Smith wrote: > ok, so PEP459 has exports: > http://www.python.org/dev/peps/pep-0459/#the-exports-extension > and distlib seems to be implementing them. > > and "exports" seem to be entry points 2.0. > > so theoretically, when PEP426/PEP459 becomes live.... > > what does that mean for setuptools-based projects? (I understand that we > might not have a clear idea now, but just trying to narrow down the > possibilities) > > 1) the setuptools "entry_points" keyword (and pkg_resources api) gets > re-implemented to understand PEP459 exports? > or > 2) "entry_points" keeps the old implementation, and possibly tools are > written to handle the old and new metadata? > or > 3) "entry_points" are stranded, and consumers have to rewrite setup.py files > to use some new keyword that understands PEP459 > or > 4) something altogether different... > > as for why I'm asking, pip itself is considering command extensions, so it's > a direct practical matter for us. > > Marcus > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From ncoghlan at gmail.com Tue Dec 31 22:20:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Jan 2014 07:20:48 +1000 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: On 1 Jan 2014 06:58, "Daniel Holth" wrote: > > It's nice to have that split up into lots of more focused documents. Yeah, I was really happy with how breaking things up into multiple extensions worked out. I'm still pondering the question of whether or not to aim for the updated installation database format (with a check for an sqlite DB in each path entry before looking directly at the metadata files) in the metadata 2.0 time frame. It's a potentially preferable approach to speeding up runtime lookups over extracting additional per-distribution data files. Maybe it's worth at least defining that as a *goal* and then deciding the time frame for when we want to get there? > The comment I have is that I think entry points sometimes have > multiple values with the same name (a list or multidict instead of a > simple dict) although I'm not sure how common that is... If the current format allows repeated keys with different values, we may need to allow lists as values in addition to strings in the PEP 459 approach. > Tools should just do automatic conversion of entry_points.txt to the > new metadata format when .dist-info is in use. Yep, as with other fields, my aim is to create a format that can be derived from the existing setuptools metadata. That way, PyPI can publish it even for projects that only upload metadata using the current setuptools formats. Cheers, Nick. > > > On Tue, Dec 31, 2013 at 3:05 PM, Marcus Smith wrote: > > ok, so PEP459 has exports: > > http://www.python.org/dev/peps/pep-0459/#the-exports-extension > > and distlib seems to be implementing them. > > > > and "exports" seem to be entry points 2.0. > > > > so theoretically, when PEP426/PEP459 becomes live.... > > > > what does that mean for setuptools-based projects? (I understand that we > > might not have a clear idea now, but just trying to narrow down the > > possibilities) > > > > 1) the setuptools "entry_points" keyword (and pkg_resources api) gets > > re-implemented to understand PEP459 exports? > > or > > 2) "entry_points" keeps the old implementation, and possibly tools are > > written to handle the old and new metadata? > > or > > 3) "entry_points" are stranded, and consumers have to rewrite setup.py files > > to use some new keyword that understands PEP459 > > or > > 4) something altogether different... > > > > as for why I'm asking, pip itself is considering command extensions, so it's > > a direct practical matter for us. > > > > Marcus > > > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Dec 31 22:35:33 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 31 Dec 2013 13:35:33 -0800 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: > > Tools should just do automatic conversion of entry_points.txt to the > > new metadata format when .dist-info is in use. > > Yep, as with other fields, my aim is to create a format that can be > derived from the existing setuptools metadata. That way, PyPI can publish > it even for projects that only upload metadata using the current setuptools > formats. > ok, but to be clear about the *usage* of the "entry_points" setup keyword... that's the user question. Should someone feel ok with adopting this now? It sounds like you're saying yes. setuptools users should assume either: a) new-style tool consumers of metadata will auto-convert old-style entry points b) setuptools will update the implementation of "entry_points" to produce be PEP459 compliant -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Tue Dec 31 23:53:07 2013 From: qwcode at gmail.com (Marcus Smith) Date: Tue, 31 Dec 2013 14:53:07 -0800 Subject: [Distutils] migration path for entry_points? In-Reply-To: References: Message-ID: let's consider 2 cases 1) someone publishes an sdist extension, SomeExtension-1.0.tar.gz up on pypi, that uses the setuptools "entry_points" keyword 2) someone publishes a wheel extension, SomeExtension-1.0.whl that has a traditional "entry_points.txt" file baked into it. and then PEP426 goes live, and pip/setuptools refactor themselves to be compliant, i.e. setuptools starts generating 2.0 metadata, and pip (via a new pkg_resources, distlib, or some other alternative) is able to process 2.0 metadata. for #1, during installation, I imagine the new setuptools would build the sdist (and any entry_point declarations) using 2.0 metadata. for #2, the pip installer would be responsible for converting "entry_points.txt" during the install (or maybe just forcing a "bdist_wheel" rebuild when it detects metadata < 2.0) On Tue, Dec 31, 2013 at 1:35 PM, Marcus Smith wrote: > > >> > Tools should just do automatic conversion of entry_points.txt to the >> > new metadata format when .dist-info is in use. >> >> Yep, as with other fields, my aim is to create a format that can be >> derived from the existing setuptools metadata. That way, PyPI can publish >> it even for projects that only upload metadata using the current setuptools >> formats. >> > ok, but to be clear about the *usage* of the "entry_points" setup > keyword... > that's the user question. Should someone feel ok with adopting this now? > It sounds like you're saying yes. > setuptools users should assume either: > a) new-style tool consumers of metadata will auto-convert old-style entry > points > b) setuptools will update the implementation of "entry_points" to > produce be PEP459 compliant > -------------- next part -------------- An HTML attachment was scrubbed... URL: