From reinout at vanrees.org Mon Nov 3 10:22:04 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 03 Nov 2014 10:22:04 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? In-Reply-To: References: <54538CED.7070205@vanrees.org> Message-ID: <5457493C.4030106@vanrees.org> On 31-10-14 18:16, Reinout van Rees wrote: > On 31-10-14 14:21, Reinout van Rees wrote: >> Ok, I tracked it down: >> https://github.com/buildout/buildout/issues/198#issuecomment-61258074 > > And I have a pull request that fixes it. > https://github.com/buildout/buildout/pull/199 > > All the buildouts that failed this morning work again with this fix. Ok, 2.2.4 is out with the fix in. A colleague came to me this morning, however, with a problem. Ouch, it isn't completely fixed yet! Turns out the `env.best_match(req, ws)` fails on anything that is installed globally with a version that doesn't match the one we want. In my colleague's case, he has nose 1.3.1 installed globally. Buildout has a pin on 1.3.4. env.best_match() returns a VersionConflict. So it seems we should ignore that too. **But**, this means buildout just gained the ability to use globally installed packages! Hurray, I don't need https://pypi.python.org/pypi/syseggrecipe anymore! I can imagine that was not quite the effect we wanted! I'll dive in a bit deeper. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity" From reinout at vanrees.org Mon Nov 3 11:28:55 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Mon, 03 Nov 2014 11:28:55 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? In-Reply-To: <5457493C.4030106@vanrees.org> References: <54538CED.7070205@vanrees.org> <5457493C.4030106@vanrees.org> Message-ID: <545758E7.2000900@vanrees.org> On 03-11-14 10:22, Reinout van Rees wrote: > Turns out the `env.best_match(req, ws)` fails on anything that is > installed globally with a version that doesn't match the one we want. > > In my colleague's case, he has nose 1.3.1 installed globally. > Buildout has a pin on 1.3.4. > env.best_match() returns a VersionConflict. > > So it seems we should ignore that too. I'm on to something. The `env.best_match()` works based on the working set. So: why is the working set filled with all the system stuff? Normally the working set starts out empty! I did a bit of extra logging and found out that buildout extensions and buildout recipes get installed with a system-wide working set. Regular requirements are installed based on a n empty working set. The problem I'm seeing is with pbp.recipe.noserunner (ancient, but handy). This has a dependency on "nose" which it finds globally with a wrong version. The other problem is with collective.recipe.sphinxbuilder, which depends on jinja2 which is installed globally in some cases ("I need ansible installed globally"). => Why does the buildout install and the extension/recipe install start out with a full-blown working set? It might be a left-over from the time when we used the globally installed setuptools. The newest bootstraps make this unnecessary, right? Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity" From reinout at vanrees.org Tue Nov 4 01:31:51 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Tue, 04 Nov 2014 01:31:51 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? In-Reply-To: <545758E7.2000900@vanrees.org> References: <54538CED.7070205@vanrees.org> <5457493C.4030106@vanrees.org> <545758E7.2000900@vanrees.org> Message-ID: On 03-11-14 11:28, Reinout van Rees wrote: > I'm on to something. https://github.com/buildout/buildout/pull/201 Multiple complications at the same time. The new buildout was stricter than the old one with constraining dependencies to the right version. For instance when installing dependencies for extensions. mr.developer depends on buildout and setuptools. Constraining them to the right version fails if the buildout was bootstrapped with a different buildout/setuptools version... Extra complication was that installing extensions happens BEFORE checking if setuptools/buildout needs upgrading/downgrading :-) The pull request also solves a problem with recipes that themselves have dependencies. pdp.recipe.noserunner depends on nose, which can be installed globally. If that's the wrong version, you get an unclear version conflict. With the pull request, buildout will ignore that conflict and instead will forcefully install the right version. This only happens when installing buildout/setuptools/extensions/recipes, though, not with the regular install/dependency handling. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity" From pthari27 at gmail.com Tue Nov 4 16:52:20 2014 From: pthari27 at gmail.com (Hari) Date: Tue, 4 Nov 2014 21:22:20 +0530 Subject: [Distutils] Help Message-ID: <024801cff847$53cd6590$fb6830b0$@com> I wrote the code for an Android app. I now need to make it as an executable app. Which utility to use? Can I use the distutils in the QPython for this purpose? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rkuska at redhat.com Tue Nov 4 14:41:23 2014 From: rkuska at redhat.com (Robert Kuska) Date: Tue, 4 Nov 2014 08:41:23 -0500 (EST) Subject: [Distutils] Nested list as value in setup argument In-Reply-To: <2071296040.530259.1415107762283.JavaMail.zimbra@redhat.com> Message-ID: <1749000868.536575.1415108483915.JavaMail.zimbra@redhat.com> Hello everyone, is there any reason why would someone enclose values in setup.py file of setup argument into nested list? Why I ask? Recently I have received bug report[0] which is affected by this feature and before I proceed to fix this I would like to know the reason why would someone do that and what perks does it bring (my plan is to flat this nested list). pkginfo; setup.py; 45 entry_points={ 46 'console_scripts': [ 47 ['pkginfo = pkginfo.commandline:main'] 48 ] 49 }, I couldn't find anything about that in setuptools docs so I hope it is ok to ask here. Regards, Robert Kuska ------------------- /home/rkuska [0] https://bitbucket.org/bkabrda/pyp2rpm/issue/21/attributeerror-list-object-has-no From eugene at sazhin.us Tue Nov 4 21:08:13 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Tue, 4 Nov 2014 15:08:13 -0500 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? Message-ID: Hello, It seems that according to the https://python-packaging-user-guide.readthedocs.org/en/latest/current.html it is more or less settled that the setuptools/pip/wheel is the way to go forward. One of the problems though is that there is plenty of packages on Pypi that are not there yet. Is there any plan to create some incentives for package maintainers to provide WHEEL files together with source archives? May be Pypi could say that starting from such date it recommends to supply the wheel with the source and then starting from such date it will require it? There is probably no better way to make the things move faster like the central repository committing to the certain way of doing things. Is that possible? Same thing could be done with python3 - ask people to provide py3 versions along with py2, and then start requiring the packages to be python3 or universal. Any opinions? Does anybody know if there is already such initiative? Thanks, Eugene From wichert at wiggy.net Tue Nov 4 22:30:33 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Tue, 4 Nov 2014 22:30:33 +0100 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: Message-ID: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> > On 04 Nov 2014, at 21:08, Evgeny Sazhin wrote: > > Hello, > > It seems that according to the > https://python-packaging-user-guide.readthedocs.org/en/latest/current.html > it is more or less settled that the setuptools/pip/wheel is the way to > go forward. > One of the problems though is that there is plenty of packages on Pypi > that are not there yet. One issue currently is that source distributions are much simpler to make: I can make a sdist without having wheel installed, and more importantly I can make a single sdist that works for both Python 2 and 3. That means I can make a release of a Python 3-compatible package without having to install Python 3 myself (or the reverse for people who are in a Python 3-only world), relying completely on a service such as Travis to run tests with Python 3. Wichert. From p.f.moore at gmail.com Wed Nov 5 00:38:25 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 4 Nov 2014 23:38:25 +0000 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: On 4 November 2014 21:30, Wichert Akkerman wrote: >> One of the problems though is that there is plenty of packages on Pypi >> that are not there yet. > > One issue currently is that source distributions are much simpler to make: I can make a sdist without having wheel installed, and more importantly I can make a single sdist that works for both Python 2 and 3. That means I can make a release of a Python 3-compatible package without having to install Python 3 myself (or the reverse for people who are in a Python 3-only world), relying completely on a service such as Travis to run tests with Python 3. If your code is single-source (i.e. doesn't need something like 2to3) then you can put in your setup.cfg [bdist_wheel] universal=1 and the wheel you build will support both Python 2 and 3, even if you built it with Python 2. Conceded you need wheel installed in order to build wheels. It's a pretty small/fast download, and you only need to install it once, but your point is true. Is it a big enough issue to justify merging wheel into (say) setuptools? I don't think so personally, but that would be the only way to avoid the need to install it separately. Paul From dholth at gmail.com Wed Nov 5 02:16:39 2014 From: dholth at gmail.com (Daniel Holth) Date: Tue, 4 Nov 2014 20:16:39 -0500 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: It is usually pretty easy to build from sdist. Wheels are convenient but I don't think they should be required. On Nov 4, 2014 6:38 PM, "Paul Moore" wrote: > On 4 November 2014 21:30, Wichert Akkerman wrote: > >> One of the problems though is that there is plenty of packages on Pypi > >> that are not there yet. > > > > One issue currently is that source distributions are much simpler to > make: I can make a sdist without having wheel installed, and more > importantly I can make a single sdist that works for both Python 2 and 3. > That means I can make a release of a Python 3-compatible package without > having to install Python 3 myself (or the reverse for people who are in a > Python 3-only world), relying completely on a service such as Travis to run > tests with Python 3. > > If your code is single-source (i.e. doesn't need something like 2to3) > then you can put in your setup.cfg > > [bdist_wheel] > universal=1 > > and the wheel you build will support both Python 2 and 3, even if you > built it with Python 2. Conceded you need wheel installed in order to > build wheels. It's a pretty small/fast download, and you only need to > install it once, but your point is true. Is it a big enough issue to > justify merging wheel into (say) setuptools? I don't think so > personally, but that would be the only way to avoid the need to > install it separately. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene at sazhin.us Wed Nov 5 00:53:57 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Tue, 4 Nov 2014 18:53:57 -0500 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: I'm also thinking about why wheel wouldn't be intergrated into standard library if this seems to be (finally) the standard? On Tue, Nov 4, 2014 at 6:38 PM, Paul Moore wrote: > On 4 November 2014 21:30, Wichert Akkerman wrote: >>> One of the problems though is that there is plenty of packages on Pypi >>> that are not there yet. >> >> One issue currently is that source distributions are much simpler to make: I can make a sdist without having wheel installed, and more importantly I can make a single sdist that works for both Python 2 and 3. That means I can make a release of a Python 3-compatible package without having to install Python 3 myself (or the reverse for people who are in a Python 3-only world), relying completely on a service such as Travis to run tests with Python 3. > > If your code is single-source (i.e. doesn't need something like 2to3) > then you can put in your setup.cfg > > [bdist_wheel] > universal=1 > > and the wheel you build will support both Python 2 and 3, even if you > built it with Python 2. Conceded you need wheel installed in order to > build wheels. It's a pretty small/fast download, and you only need to > install it once, but your point is true. Is it a big enough issue to > justify merging wheel into (say) setuptools? I don't think so > personally, but that would be the only way to avoid the need to > install it separately. > > Paul From eugene at sazhin.us Wed Nov 5 04:28:11 2014 From: eugene at sazhin.us (Evgeny Sazhin) Date: Tue, 4 Nov 2014 22:28:11 -0500 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: My experience so far tells me otherwise. Our of 7 or so libraries that I tried to convert to wheel files that salt stack depends on only 2 were using setuptools, others were using distutils and had sometimes quite big setup.py files and were not compiling out of the box, and frankly I have no idea how I can create a wheel from a setup.py file that is taking 10 screens to scroll and not using the packager that is supporting wheel and has dependencies on some C stuff. I think the notion of sdist being a magic thing that everybody should be able to make anything from doesn't actually work for end users. I think sdist has it's place but the guidance that people need should be quite explicit about how and what to do to end the confusion. Standard is standard and should be set in stone as such. Otherwise everybody keeps wondering around and doing whatever they see fit and effectively creating a mess without the ability to sync on common ground. like java jars are the standard way of distributing the libs, same thing should be with python packages. Everybody should know that wheel is that thing, nothing else. Doesn't it make sense? Thanks! On Nov 4, 2014 8:16 PM, "Daniel Holth" wrote: > It is usually pretty easy to build from sdist. Wheels are convenient but I > don't think they should be required. > On Nov 4, 2014 6:38 PM, "Paul Moore" wrote: > >> On 4 November 2014 21:30, Wichert Akkerman wrote: >> >> One of the problems though is that there is plenty of packages on Pypi >> >> that are not there yet. >> > >> > One issue currently is that source distributions are much simpler to >> make: I can make a sdist without having wheel installed, and more >> importantly I can make a single sdist that works for both Python 2 and 3. >> That means I can make a release of a Python 3-compatible package without >> having to install Python 3 myself (or the reverse for people who are in a >> Python 3-only world), relying completely on a service such as Travis to run >> tests with Python 3. >> >> If your code is single-source (i.e. doesn't need something like 2to3) >> then you can put in your setup.cfg >> >> [bdist_wheel] >> universal=1 >> >> and the wheel you build will support both Python 2 and 3, even if you >> built it with Python 2. Conceded you need wheel installed in order to >> build wheels. It's a pretty small/fast download, and you only need to >> install it once, but your point is true. Is it a big enough issue to >> justify merging wheel into (say) setuptools? I don't think so >> personally, but that would be the only way to avoid the need to >> install it separately. >> >> Paul >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Nov 5 06:06:36 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 5 Nov 2014 00:06:36 -0500 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: Bad setup.py is a different issue. I didn't have a solution for that so I did a binary package format instead. I wish there was more innovation on the build side. We should be building sdists that can make wheels with no DistUtils or setuptools. "pip wheel ." is sometimes easier than setup.py bdist_wheel, like any pip install it will force setuptools to be used. On Nov 4, 2014 10:28 PM, "Evgeny Sazhin" wrote: > My experience so far tells me otherwise. Our of 7 or so libraries that I > tried to convert to wheel files that salt stack depends on only 2 were > using setuptools, others were using distutils and had sometimes quite big > setup.py files and were not compiling out of the box, and frankly I have no > idea how I can create a wheel from a setup.py file that is taking 10 > screens to scroll and not using the packager that is supporting wheel and > has dependencies on some C stuff. > > I think the notion of sdist being a magic thing that everybody should be > able to make anything from doesn't actually work for end users. > > I think sdist has it's place but the guidance that people need should be > quite explicit about how and what to do to end the confusion. > > Standard is standard and should be set in stone as such. Otherwise > everybody keeps wondering around and doing whatever they see fit and > effectively creating a mess without the ability to sync on common ground. > like java jars are the standard way of distributing the libs, same thing > should be with python packages. Everybody should know that wheel is that > thing, nothing else. Doesn't it make sense? > > Thanks! > On Nov 4, 2014 8:16 PM, "Daniel Holth" wrote: > >> It is usually pretty easy to build from sdist. Wheels are convenient but >> I don't think they should be required. >> On Nov 4, 2014 6:38 PM, "Paul Moore" wrote: >> >>> On 4 November 2014 21:30, Wichert Akkerman wrote: >>> >> One of the problems though is that there is plenty of packages on Pypi >>> >> that are not there yet. >>> > >>> > One issue currently is that source distributions are much simpler to >>> make: I can make a sdist without having wheel installed, and more >>> importantly I can make a single sdist that works for both Python 2 and 3. >>> That means I can make a release of a Python 3-compatible package without >>> having to install Python 3 myself (or the reverse for people who are in a >>> Python 3-only world), relying completely on a service such as Travis to run >>> tests with Python 3. >>> >>> If your code is single-source (i.e. doesn't need something like 2to3) >>> then you can put in your setup.cfg >>> >>> [bdist_wheel] >>> universal=1 >>> >>> and the wheel you build will support both Python 2 and 3, even if you >>> built it with Python 2. Conceded you need wheel installed in order to >>> build wheels. It's a pretty small/fast download, and you only need to >>> install it once, but your point is true. Is it a big enough issue to >>> justify merging wheel into (say) setuptools? I don't think so >>> personally, but that would be the only way to avoid the need to >>> install it separately. >>> >>> Paul >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 5 19:24:00 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 05 Nov 2014 10:24:00 -0800 Subject: [Distutils] pip not grabbing latest PyPI version Message-ID: <545A6B40.6090900@stoneleaf.us> I have four packages on PyPI: antipathy, dbf, pandaemonium, and scription. `pip install --upgrade` works for three of them, but for scription it continually grabs an older release (0.53, I think). Any ideas what might be wrong? -- ~Ethan~ From chris.barker at noaa.gov Wed Nov 5 18:31:00 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 5 Nov 2014 09:31:00 -0800 Subject: [Distutils] Pypi to enforce wheel, python3 - is it possible? In-Reply-To: References: <52B56B14-314C-4A42-AB16-826678F30163@wiggy.net> Message-ID: no, wheels should not be required -- encourage, absolutely, but required, no. > My experience so far tells me otherwise. Our of 7 or so libraries that I > tried to convert to wheel files that salt stack depends on only 2 were > using setuptools, others were using distutils and had sometimes quite big > setup.py files and were not compiling out of the box, and frankly I have no > idea how I can create a wheel from a setup.py file that is taking 10 > screens to scroll and not using the packager that is supporting wheel and > has dependencies on some C stuff. > Exactly -- some stuff is hard to build -- period, end of story. Ideally, everyone would make binary wheels for al platforms for these, but even that is next to impossible universally (see the conda package manger -- it's been developed for a reason). So should we say: if you haven't made a wheel out of it, it can't be on PyPi? that seems like abad idea overall. Also -- if wheels were required, what versions/platforms etc. would be required? That's a rabbit hole we should avoid! > It is usually pretty easy to build from sdist. Wheels are convenient but I >> don't think they should be required. >> > Actually, if it's easy to build from sdist, then it's easy to make wheels ;-) -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Thu Nov 6 09:29:20 2014 From: marius at pov.lt (Marius Gedminas) Date: Thu, 6 Nov 2014 10:29:20 +0200 Subject: [Distutils] pip not grabbing latest PyPI version In-Reply-To: <545A6B40.6090900@stoneleaf.us> References: <545A6B40.6090900@stoneleaf.us> Message-ID: <20141106082920.GA31523@fridge.pov.lt> On Wed, Nov 05, 2014 at 10:24:00AM -0800, Ethan Furman wrote: > I have four packages on PyPI: antipathy, dbf, pandaemonium, and scription. > > `pip install --upgrade` works for three of them, but for scription it continually grabs an older release (0.53, I think). I cannot reproduce pip install scription fetches 0.70.4 for me. Marius Gedminas -- Never attribute to malloc that which can be adequately explained by stupidity. -- From the .sig of joerg at raleigh.ibm.com (Joerg Pommnitz) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Thu Nov 6 10:15:15 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 6 Nov 2014 19:15:15 +1000 Subject: [Distutils] Help In-Reply-To: <024801cff847$53cd6590$fb6830b0$@com> References: <024801cff847$53cd6590$fb6830b0$@com> Message-ID: On 5 Nov 2014 02:22, "Hari" wrote: > > I wrote the code for an Android app. I now need to make it as an executable app. Which utility to use? Can I use the distutils in the QPython for this purpose? If you mean a Python Qt application, then you may want pyqtdeploy ( http://pyqt.sourceforge.net/Docs/pyqtdeploy/introduction.html) Otherwise, I'm not clear on what you mean by "QPython". Regards, Nick. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p at 2014.dobrogost.net Thu Nov 6 11:26:35 2014 From: p at 2014.dobrogost.net (Piotr Dobrogost) Date: Thu, 6 Nov 2014 11:26:35 +0100 Subject: [Distutils] pip not grabbing latest PyPI version In-Reply-To: <545A6B40.6090900@stoneleaf.us> References: <545A6B40.6090900@stoneleaf.us> Message-ID: On Wed, Nov 5, 2014 at 7:24 PM, Ethan Furman wrote: > I have four packages on PyPI: antipathy, dbf, pandaemonium, and scription. > > `pip install --upgrade` works for three of them, but for scription it > continually grabs an older release (0.53, I think). > > Any ideas what might be wrong? Maybe this would help ? http://stackoverflow.com/questions/14617136/why-is-pip-installing-an-old-version-of-my-package Regards, Piotr Dobrogost From pmiscml at gmail.com Tue Nov 4 22:15:01 2014 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Tue, 4 Nov 2014 23:15:01 +0200 Subject: [Distutils] Efficiently supporting multiple platforms in distribution packages In-Reply-To: References: <20141027171512.339f3656@x230> Message-ID: <20141104231501.0198e45f@x230> Hello, On Mon, 27 Oct 2014 15:49:21 +0000 Paul Moore wrote: > On 27 October 2014 15:15, Paul Sokolovsky wrote: > >> For a source distribution, you could play clever games in setup.py > >> to put the right file in place, with the right name. But that's > >> messy and it means that if you distribute wheels (not that there's > >> much point in doing so) you need separate wheels for 2.6-, 2.7 and > >> 3.3+. > > > > Ok, so are there guidelines, best practices or at least example(s) > > how to do that? I pretty much would like to avoid inventing my own > > "clever games" to achieve that. > > Personally, my guideline would be "don't do that". We're trying to > move away from having complex code in setup.py, to a more nearly > declarative solution. It's great to hear there's move in declarative direction. And all the above is actually the reason I wrote "I don't want to invent my own way to do it". What I'd like to have is such declarative/data-driven way to select which files to install, based on some predefined (alternatively, user-defined) criteria. As current setuptools work in imperative manner, I essentially asked 1) how to code up conditional file installation with setuptools; 2) how to set my "custom" metadata/structure in such a way that it would be compatible with current community direction/best practices towards future fully declarative solution. Thanks for the code in the other message, it gives good hints for question 1. I still could use some guidelines on question 2 (or alternatively, as you write below, maybe you/other distutil-sig developers could use it when designing future solution). Let me give one example of possible many how these could be structured: So, in my src dir, I could have: os.py PLATFORM-macosx/os.py PLATFORM-pyboard/os.py Installer could see a special directory names starting with a set of predefined property names (uppercased to avoid easy clashing with user dirs). In this case, PLATFORM means sys.platform. If substring after hyphen matches its value, corresponding file is installed, otherwise file in the main dir is used as a fallback. So, would the schedule above be along the lines of your thinking how that should be done, or you're immediately see bunch of issues with it (I sure do - it's easy on packager's side, by not so easy on tool's side to figure what packager wanted, so more disambiguation surely would be required)? Then, would you have suggestion for better structure? If the intent of the above is still not clear enough, let me put it this way: with MicroPython, we won't be able to use pip in self-hosting mode (for various reasons), so we'd need to develop own lightweight package manager. It for sure would use declarative approach. So, we're looking for what kind of metadata and approach we should take to stay with general Python community's outlook of this? (And we want our packages to be still installable with current and future pip of course.) Thanks! > The above approach pretty much directly goes > against that. I posted code later in the same thread, but I still > don't think people should be doing it. > > I see the issue you and Ethan have, though. It should be considered in > the context of Metadata 2.0 and things like that, so we make sure the > use case is covered. > > Paul -- Best regards, Paul mailto:pmiscml at gmail.com From techtonik at gmail.com Thu Nov 6 10:04:04 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Thu, 6 Nov 2014 12:04:04 +0300 Subject: [Distutils] pip not grabbing latest PyPI version In-Reply-To: <545A6B40.6090900@stoneleaf.us> References: <545A6B40.6090900@stoneleaf.us> Message-ID: On Wed, Nov 5, 2014 at 9:24 PM, Ethan Furman wrote: > I have four packages on PyPI: antipathy, dbf, pandaemonium, and scription. > > `pip install --upgrade` works for three of them, but for scription it > continually grabs an older release (0.53, I think). > > Any ideas what might be wrong? - CDN - proxy - not PyPI as package repository - --pre version - pip bug -- anatoly t. From ethan at stoneleaf.us Thu Nov 6 15:53:44 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 06 Nov 2014 06:53:44 -0800 Subject: [Distutils] pip not grabbing latest PyPI version In-Reply-To: References: <545A6B40.6090900@stoneleaf.us> Message-ID: <545B8B78.4050604@stoneleaf.us> [redirecting to list] On 11/05/2014 10:30 AM, Ian Cordasco wrote: > On Nov 5, 2014 12:24 PM, "Ethan Furman" wrote: >> >> I have four packages on PyPI: antipathy, dbf, pandaemonium, and scription. >> >> `pip install --upgrade` works for three of them, but for scription it continually grabs an older release (0.53, I think). >> >> Any ideas what might be wrong? > > Is the newest version a pre-release? No, but that was the clue I need, thanks! 0.53.0 is a higher version than 0.7.0. Doh. Changing that to 0.70.1 now... Thanks for the help! -- ~Ethan~ From p.f.moore at gmail.com Fri Nov 7 16:46:36 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 7 Nov 2014 15:46:36 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? Message-ID: I'm in the process of developing an automated solution to allow users to quickly set up a Windows box so that it can be used to compile Python extensions and build wheels. While it can obviously be used by Windows developers who want to quickly set up a box, my main target is Unix developers who want to provide wheels for Windows users. To that end, I'd like to get an idea of what sort of access to Windows a typical Unix developer would have. I'm particularly interested in whether Windows XP/Vista is still in use, and whether you're likely to already have Python and/or any development tools installed. Ideally, a clean Windows 7 or later virtual machine is the best environment, but I don't know if it's reasonable to assume that. Another alternative is to have an Amazon EC2 AMI prebuilt, and users can just create an instance based on it. That seems pretty easy to do from my perspective but I don't know if the connectivity process (remote desktop) is a problem for Unix developers. Any feedback would be extremely useful. I'm at a point where I can pretty easily set up any of these options, but if they don't turn out to actually be usable by the target audience, it's a bit of a waste of time! :-) Thanks, Paul From donald at stufft.io Fri Nov 7 16:49:26 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 7 Nov 2014 10:49:26 -0500 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <9DE4D646-1948-4FF9-9F88-68755493B43B@stufft.io> > On Nov 7, 2014, at 10:46 AM, Paul Moore wrote: > > I'm in the process of developing an automated solution to allow users > to quickly set up a Windows box so that it can be used to compile > Python extensions and build wheels. While it can obviously be used by > Windows developers who want to quickly set up a box, my main target is > Unix developers who want to provide wheels for Windows users. > > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. I'm particularly interested in > whether Windows XP/Vista is still in use, and whether you're likely to > already have Python and/or any development tools installed. Ideally, a > clean Windows 7 or later virtual machine is the best environment, but > I don't know if it's reasonable to assume that. > > Another alternative is to have an Amazon EC2 AMI prebuilt, and users > can just create an instance based on it. That seems pretty easy to do > from my perspective but I don't know if the connectivity process > (remote desktop) is a problem for Unix developers. > > Any feedback would be extremely useful. I'm at a point where I can > pretty easily set up any of these options, but if they don't turn out > to actually be usable by the target audience, it's a bit of a waste of > time! :-) > > Thanks, > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig As an *nix user I have a Windows 7 VM on my OS X machine that I can also dual boot into which I mostly use for playing games that won?t play on my OS X box natively. It does not have Python or any development tooling installed on it. I also have access to the cloud(tm) which is where I normally spin up a whatever-the-most-recent-looking-name Windows Server. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Fri Nov 7 17:09:54 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 7 Nov 2014 17:09:54 +0100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? References: Message-ID: <20141107170954.7a403ada@fsol> On Fri, 7 Nov 2014 15:46:36 +0000 Paul Moore wrote: > I'm in the process of developing an automated solution to allow users > to quickly set up a Windows box so that it can be used to compile > Python extensions and build wheels. While it can obviously be used by > Windows developers who want to quickly set up a box, my main target is > Unix developers who want to provide wheels for Windows users. > > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. I'm particularly interested in > whether Windows XP/Vista is still in use, and whether you're likely to > already have Python and/or any development tools installed. Ideally, a > clean Windows 7 or later virtual machine is the best environment, but > I don't know if it's reasonable to assume that. I use a Windows 7 VM with SP1. All running under KVM with virt-manager. Regards Antoine. From wichert at wiggy.net Fri Nov 7 17:17:18 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Fri, 7 Nov 2014 17:17:18 +0100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <4678CD6F-D645-4C04-85E5-1D94E72A590D@wiggy.net> > On 07 Nov 2014, at 16:46, Paul Moore wrote: > > I'm in the process of developing an automated solution to allow users > to quickly set up a Windows box so that it can be used to compile > Python extensions and build wheels. While it can obviously be used by > Windows developers who want to quickly set up a box, my main target is > Unix developers who want to provide wheels for Windows users. > > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. In my case: none. The only form of Windows I have are VMs I grab from modern.ie to test things with various IE versions. Those are all throw-away instances that are never used for anything other than IE testing. Wichert. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marius at pov.lt Fri Nov 7 17:26:53 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 7 Nov 2014 18:26:53 +0200 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <20141107162653.GA11638@fridge.pov.lt> On Fri, Nov 07, 2014 at 03:46:36PM +0000, Paul Moore wrote: > I'm in the process of developing an automated solution to allow users > to quickly set up a Windows box so that it can be used to compile > Python extensions and build wheels. While it can obviously be used by > Windows developers who want to quickly set up a box, my main target is > Unix developers who want to provide wheels for Windows users. > > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. I'm particularly interested in > whether Windows XP/Vista is still in use, and whether you're likely to > already have Python and/or any development tools installed. Ideally, a > clean Windows 7 or later virtual machine is the best environment, but > I don't know if it's reasonable to assume that. > > Another alternative is to have an Amazon EC2 AMI prebuilt, and users > can just create an instance based on it. That seems pretty easy to do > from my perspective but I don't know if the connectivity process > (remote desktop) is a problem for Unix developers. > > Any feedback would be extremely useful. I don't maintain any Python packages with C extensions, so I'm not sure my feedback is useful. Nevertheless: I have a cloud VM running Windows Server WhicheverWasTheHighestNumberAtTheTime with no C compilers on it. (I don't think the bits of Mingw that come with Git for Windows include gcc.) I have all the relevant versions of Python installed on it, with added setuptools and pip in each. IIRC I also installed tox and virtualenv into their site packages. I use this VM as a Jenkins slave to run tests of various Python packages. Some of them need binaries, and I rely on package maintainers to upload Windows wheels to PyPI. Since *of course* they don't all do that, so I have to maintain a set of wheels automatically converted from .exe and .egg installers with https://github.com/mgedmin/wheelwright. Anything that makes it easier for package maintainers build Windows wheels would be very welcome! Marius Gedminas -- For every complex problem, there is a solution that is simple, neat, and wrong -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From ben+python at benfinney.id.au Fri Nov 7 17:52:02 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 08 Nov 2014 03:52:02 +1100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? References: Message-ID: <851tpf54y5.fsf@benfinney.id.au> Paul Moore writes: > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. [?] Ideally, a clean Windows 7 or > later virtual machine is the best environment, but I don't know if > it's reasonable to assume that. It's difficult to say what ?a typical Unix developer? is. But a significant use case is going to be ?no legal access to any MS Windows instance?. The restrictions of the license terms make MS Windows an unacceptable risk on any machine I'm responsible for. It has been many years since I've even had a colleague who has a MS Windows instance, and I am not sure where I'd go for one if the need arose. If I was required to provide packages for MS Windows, the only viable solutions would be those that don't involve me obtaining an MS Windows instance myself. -- \ ?Ignorance more frequently begets confidence than does | `\ knowledge.? ?Charles Darwin, _The Descent of Man_, 1871 | _o__) | Ben Finney From Steve.Dower at microsoft.com Fri Nov 7 18:05:47 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 7 Nov 2014 17:05:47 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <851tpf54y5.fsf@benfinney.id.au> References: <851tpf54y5.fsf@benfinney.id.au> Message-ID: <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> Ben Finney wrote: > Paul Moore writes: > >> To that end, I'd like to get an idea of what sort of access to Windows >> a typical Unix developer would have. [?] Ideally, a clean Windows 7 or >> later virtual machine is the best environment, but I don't know if >> it's reasonable to assume that. > > It's difficult to say what ?a typical Unix developer? is. But a significant use > case is going to be ?no legal access to any MS Windows instance?. > > The restrictions of the license terms make MS Windows an unacceptable risk on > any machine I'm responsible for. Just out of interest, which restrictions would those be? I may be able to raise them with one of our lawyers and get some clarification. > It has been many years since I've even had a colleague who has a MS Windows > instance, and I am not sure where I'd go for one if the need arose. > > If I was required to provide packages for MS Windows, the only viable solutions > would be those that don't involve me obtaining an MS Windows instance myself. Does this prevent you from creating a VM on a cloud provider on your own account? As far as Microsoft Azure is concerned, this is well within the license restrictions (at least for Windows Server right now), and all providers giving you access to Windows should be bundling in a license fee, which makes it about as legit as possible. Simply giving you "share time" on someone else's copy of Windows is much more of a grey area as far as licensing is concerned. If the licensing is a real issue, I'm in a position where I can have a positive impact on fixing it, so any info you can provide me (on- or off-list) about your concerns is valuable. Cheers, Steve From p.f.moore at gmail.com Fri Nov 7 18:13:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 7 Nov 2014 17:13:22 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <851tpf54y5.fsf@benfinney.id.au> References: <851tpf54y5.fsf@benfinney.id.au> Message-ID: On 7 November 2014 16:52, Ben Finney wrote: > If I was required to provide packages for MS Windows, the only viable > solutions would be those that don't involve me obtaining an MS Windows > instance myself. For that usage, an Amazon EC2 AMI sounds ideal, as the license costs are covered by the AWS costs (which are zero, if you're on the free usage tier). Paul From ben+python at benfinney.id.au Fri Nov 7 18:17:29 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 08 Nov 2014 04:17:29 +1100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? References: <851tpf54y5.fsf@benfinney.id.au> Message-ID: <85wq773p7a.fsf@benfinney.id.au> Paul Moore writes: > On 7 November 2014 16:52, Ben Finney wrote: > > If I was required to provide packages for MS Windows, the only viable > > solutions would be those that don't involve me obtaining an MS Windows > > instance myself. > > For that usage [?] the license costs [?] I didn't mention monetary costs at all. My understanding is that changing the cost doesn't in any way affect the terms of the license one is bound by. -- \ ?I have never imputed to Nature a purpose or a goal, or | `\ anything that could be understood as anthropomorphic.? ?Albert | _o__) Einstein, unsent letter, 1955 | Ben Finney From p.f.moore at gmail.com Fri Nov 7 18:21:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 7 Nov 2014 17:21:15 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <85wq773p7a.fsf@benfinney.id.au> References: <851tpf54y5.fsf@benfinney.id.au> <85wq773p7a.fsf@benfinney.id.au> Message-ID: On 7 November 2014 17:17, Ben Finney wrote: > Paul Moore writes: > >> On 7 November 2014 16:52, Ben Finney wrote: >> > If I was required to provide packages for MS Windows, the only viable >> > solutions would be those that don't involve me obtaining an MS Windows >> > instance myself. >> >> For that usage [?] the license costs [?] > > I didn't mention monetary costs at all. My understanding is that > changing the cost doesn't in any way affect the terms of the license one > is bound by. Sorry, I misunderstood you. As Steve said, it would be necessary to understand the restrictions you're working under to be able to comment. Paul From ben+python at benfinney.id.au Fri Nov 7 18:42:51 2014 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 08 Nov 2014 04:42:51 +1100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? References: <851tpf54y5.fsf@benfinney.id.au> <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <85sihu52lg.fsf@benfinney.id.au> Steve Dower writes: > Ben Finney wrote: > > The restrictions of the license terms make MS Windows an > > unacceptable risk on any machine I'm responsible for. > > Just out of interest, which restrictions would those be? It has been a long time since I bothered to read any of the numerous license texts from Microsoft, so I can't cite specific clauses. From memory, unacceptable restrictions include: * Restricting the instance to specific hardware, instead of leaving it up to the recipient to run the work they paid for on any hardware they choose. * Forbidding reverse-engineering of the OS to see how it behaves. * Forbidding collaboration with other recipients to discover how the OS behaves. * Refusal to disclose the source code for the running OS to the recipient. * Forbidding the recipient from getting their choice of vendor to make improvements to the OS and collaborate with other recipients on the improvements. * Arrogating control of the running OS to a party other than the license recipient, including the ability to (at Microsoft's sole discretion) deny applications to run, and to disable features of the OS. * Arrogating data collection to Microsoft and undisclosed third parties, tracking broad classes of activity on the OS and sending the logs to a server not of the recipient's choosing. > Does this prevent you from creating a VM on a cloud provider on your > own account? If I need to accept restrictions such as the above, I don't see that the location of the instance (nor the fees charged) has any affect on these concerns. The risks discussed above are not mitigated. > If the licensing is a real issue, I'm in a position where I can have a > positive impact on fixing it, so any info you can provide me (on- or > off-list) about your concerns is valuable. Thank you for this offer, I am glad to see willingness expressed to solve these restrictions. I hope you can achieve software freedom for all recipients of Microsoft operating systems. Until then, the risk is too great to anyone to whom I have professional responsibilities, and my advice must continue to be that they avoid accepting such restrictions. -- \ ?Simplicity is prerequisite for reliability.? ?Edsger W. | `\ Dijkstra | _o__) | Ben Finney From p.f.moore at gmail.com Fri Nov 7 19:08:12 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 7 Nov 2014 18:08:12 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <85sihu52lg.fsf@benfinney.id.au> References: <851tpf54y5.fsf@benfinney.id.au> <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> <85sihu52lg.fsf@benfinney.id.au> Message-ID: On 7 November 2014 17:42, Ben Finney wrote: >> Does this prevent you from creating a VM on a cloud provider on your >> own account? > > If I need to accept restrictions such as the above, I don't see that the > location of the instance (nor the fees charged) has any affect on these > concerns. The risks discussed above are not mitigated. Thanks for the clarification. Given what you say, I don't see any way that I can offer a solution you'd be willing to accept - I suspect the only viable option for you would be support for cross-compilation using mingw/ggg, which I'm not able to offer. For now, I guess, that simply means I'll have to consider you (and anyone else for whom even running a Windows system is unacceptable) outside of my target audience. Paul From christian at python.org Fri Nov 7 20:11:35 2014 From: christian at python.org (Christian Heimes) Date: Fri, 07 Nov 2014 20:11:35 +0100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <545D1967.2090600@python.org> On 07.11.2014 16:46, Paul Moore wrote: > Any feedback would be extremely useful. I'm at a point where I can > pretty easily set up any of these options, but if they don't turn out > to actually be usable by the target audience, it's a bit of a waste of > time! :-) I'm hosting two Windows 7 Ultimate 64bit instances with VirtualBox, one my laptop and the other on my work station. Each box has two Visual Studio installations for Python 2.7 and 3.3+. All licenses are sponsored by Microsoft (MSDN subscription). Christian From ethan at stoneleaf.us Fri Nov 7 20:18:59 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 07 Nov 2014 11:18:59 -0800 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <545D1B23.2010102@stoneleaf.us> I have a couple languishing laptops that still have XP on them. I have one desktop with Win7, used primarily for unittesting my modules (no dev tools beyond Python, although I should get VS on it at some point). -- ~Ethan~ From jjhelmus at gmail.com Fri Nov 7 17:15:04 2014 From: jjhelmus at gmail.com (Jonathan Helmus) Date: Fri, 07 Nov 2014 10:15:04 -0600 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <545CF008.1020408@gmail.com> On 11/07/2014 09:46 AM, Paul Moore wrote: > I'm in the process of developing an automated solution to allow users > to quickly set up a Windows box so that it can be used to compile > Python extensions and build wheels. While it can obviously be used by > Windows developers who want to quickly set up a box, my main target is > Unix developers who want to provide wheels for Windows users. > > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. I'm particularly interested in > whether Windows XP/Vista is still in use, and whether you're likely to > already have Python and/or any development tools installed. Ideally, a > clean Windows 7 or later virtual machine is the best environment, but > I don't know if it's reasonable to assume that. > > Another alternative is to have an Amazon EC2 AMI prebuilt, and users > can just create an instance based on it. That seems pretty easy to do > from my perspective but I don't know if the connectivity process > (remote desktop) is a problem for Unix developers. > > Any feedback would be extremely useful. I'm at a point where I can > pretty easily set up any of these options, but if they don't turn out > to actually be usable by the target audience, it's a bit of a waste of > time! :-) > > Thanks, > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Paul, This sounds like a very useful project. I do most my development in Linux or OS X but occasionally spin up a Windows VM locally or in the cloud to produce some Python wheels or conda packages. The one annoyance I find is that I need to use an RDP client to connect to the instance as opposed to using ssh as I do when connecting to a Unix machine. If the VM could support ssh connections this would reduce this burden. I am not familiar enough with Windows to know how feasible this is but it would allow Unix users to use the tools they are likely already familiar with. - Jonathan Helmus From pmiscml at gmail.com Sat Nov 8 22:04:30 2014 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Sat, 8 Nov 2014 23:04:30 +0200 Subject: [Distutils] pypi.python.org forces SSL? Message-ID: <20141108230430.461fe2c1@x230> Hello, I see that nowadays trying to access http://pypi.python.org (including any URL beneath it) unconditionally redirects to https://pypi.python.org. I'm trying to access it using system which doesn't have SSL support and cannot easily have (embedded, size constraints). Is there any way to access PyPI metadata/tarballs using plain HTTP? Thanks, Paul mailto:pmiscml at gmail.com From p.f.moore at gmail.com Sun Nov 9 12:13:36 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 11:13:36 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <545CF008.1020408@gmail.com> References: <545CF008.1020408@gmail.com> Message-ID: On 7 November 2014 16:15, Jonathan Helmus wrote: > This sounds like a very useful project. I do most my development in Linux > or OS X but occasionally spin up a Windows VM locally or in the cloud to > produce some Python wheels or conda packages. The one annoyance I find is > that I need to use an RDP client to connect to the instance as opposed to > using ssh as I do when connecting to a Unix machine. If the VM could > support ssh connections this would reduce this burden. I am not familiar > enough with Windows to know how feasible this is but it would allow Unix > users to use the tools they are likely already familiar with. Thanks, that's very useful feedback. I agree, the need for RDP is very Windows-specific - I don't know how common RDP tools are for Unix, but I can easily imagine it's a nuisance to set one up. I'm not aware of any ssh implementations for Windows (other than cygwin, which probably has other issues) but I'll see what I can find. Paul From vinay_sajip at yahoo.co.uk Sun Nov 9 13:13:23 2014 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 9 Nov 2014 12:13:23 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> Message-ID: <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> > Thanks, that's very useful feedback. I agree, the need for RDP is very > Windows-specific - I don't know how common RDP tools are for Unix, but Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access Windows machines via RDP, and it seems fairly stable. There are alternative tools available (such as remmina). Regards, Vinay Sajip From ncoghlan at gmail.com Sun Nov 9 13:28:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 9 Nov 2014 22:28:51 +1000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 9 Nov 2014 22:16, "Vinay Sajip" wrote: > > > Thanks, that's very useful feedback. I agree, the need for RDP is very > > > Windows-specific - I don't know how common RDP tools are for Unix, but > > > > > Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access Windows machines via RDP, and it seems fairly stable. There are alternative tools available (such as remmina). For automated scripting from a *nix host, it's also worth noting that Ansible supports remote control of Windows systems, and the underlying Python libraries for that connectivity are also open source (they talk to the native Windows remote control interfaces, so they don't need anything special on the target system). Cheers, Nick. > > Regards, > > Vinay Sajip > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Nov 9 13:28:12 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 12:28:12 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <545F5C57.5020406@timgolden.me.uk> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> <545F5C57.5020406@timgolden.me.uk> Message-ID: On 9 November 2014 12:21, Tim Golden wrote: > I think the OP was speaking not so much about having the technical > wherewithal to use RDP but rather about the experience of RDP vs SSH. That was certainly my understanding. The key issue for me is to try to make the process of just running "pip wheel myproject" or "pip wheel git+https://github.com/me/myproject" as simple and painless as possible for people without Windows experience. That's somewhat optimistic, because if the command fails with an error, the developer is still going to need to work out how to debug why the code isn't portable, etc. But that's a whole different situation, and well out of scope. > The > difficulty is that Windows doesn't really "think" in ssh. I believe there > are (third-party) mechanisms to provide ssh-like access, but I don't know > how successful they really are. Yeah, that's where things like cygwin probably won't work well, because you don't get the "normal" Windows environment. But it might be possible - after all, see above - it's really only a few simple commands we need to support. Paul From ncoghlan at gmail.com Sun Nov 9 13:40:36 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 9 Nov 2014 22:40:36 +1000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 9 Nov 2014 22:28, "Nick Coghlan" wrote: > > > On 9 Nov 2014 22:16, "Vinay Sajip" wrote: > > > > > Thanks, that's very useful feedback. I agree, the need for RDP is very > > > > > Windows-specific - I don't know how common RDP tools are for Unix, but > > > > > > > > > > Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access Windows machines via RDP, and it seems fairly stable. There are alternative tools available (such as remmina). > > For automated scripting from a *nix host, it's also worth noting that Ansible supports remote control of Windows systems, and the underlying Python libraries for that connectivity are also open source (they talk to the native Windows remote control interfaces, so they don't need anything special on the target system). Oops, forgot the link: http://docs.ansible.com/intro_windows.html Underlying library: https://pypi.python.org/pypi/pywinrm Cheers, Nick. > > Cheers, > Nick. > > > > > Regards, > > > > Vinay Sajip > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Nov 9 13:40:55 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 12:40:55 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 9 November 2014 12:28, Nick Coghlan wrote: > For automated scripting from a *nix host, it's also worth noting that > Ansible supports remote control of Windows systems, and the underlying > Python libraries for that connectivity are also open source (they talk to > the native Windows remote control interfaces, so they don't need anything > special on the target system). I would *love* to be able to set up something like that (I believe Salt has Windows support as well - I hadn't realised Ansible did). But that would involve me learning a big chunk of Unix sysop stuff, as well as having a number of machines to play with. Not enough hours in the day, unfortunately. Paul From cournape at gmail.com Sun Nov 9 14:04:06 2014 From: cournape at gmail.com (David Cournapeau) Date: Sun, 9 Nov 2014 13:04:06 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: Regarding winrm, note that it can be fairly complicated to set it up on the windows side. I can confirm it works well, though. We're using an internal hack of fabric to communicate from linux to windows through winrm at work, and it does the job. David On Sun, Nov 9, 2014 at 12:40 PM, Nick Coghlan wrote: > > On 9 Nov 2014 22:28, "Nick Coghlan" wrote: > > > > > > On 9 Nov 2014 22:16, "Vinay Sajip" wrote: > > > > > > > Thanks, that's very useful feedback. I agree, the need for RDP is > very > > > > > > > Windows-specific - I don't know how common RDP tools are for Unix, > but > > > > > > > > > > > > > > > Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access > Windows machines via RDP, and it seems fairly stable. There are alternative > tools available (such as remmina). > > > > For automated scripting from a *nix host, it's also worth noting that > Ansible supports remote control of Windows systems, and the underlying > Python libraries for that connectivity are also open source (they talk to > the native Windows remote control interfaces, so they don't need anything > special on the target system). > > Oops, forgot the link: http://docs.ansible.com/intro_windows.html > > Underlying library: https://pypi.python.org/pypi/pywinrm > > Cheers, > Nick. > > > > > Cheers, > > Nick. > > > > > > > > Regards, > > > > > > Vinay Sajip > > > _______________________________________________ > > > Distutils-SIG maillist - Distutils-SIG at python.org > > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Nov 9 14:04:07 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 13:04:07 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 9 November 2014 12:40, Nick Coghlan wrote: > Underlying library: https://pypi.python.org/pypi/pywinrm Wow - that sounds awesome. Not that I can get it working yet, looks like WinRM might need some setting up (and/or VirtualBox networking is getting in the way :-() But that is definitely something I'll be looking at. Thanks, Paul From p.f.moore at gmail.com Sun Nov 9 15:17:05 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 14:17:05 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: On 9 November 2014 13:04, David Cournapeau wrote: > Regarding winrm, note that it can be fairly complicated to set it up on the > windows side. For my simple test, "winrm quickconfig" worked OK. Having said all this, though, I'm not sure that hiding the Windows environment this thoroughly is productive. If you just want to build wheels, and you don't intend to be debugging Windows issues on the Windows box, then Appveyor is probably more appropriate (see https://packaging.python.org/en/latest/appveyor.html). If you want an actual Windows environment, surely you expect to need to log onto it? Paul From donald at stufft.io Sun Nov 9 15:44:05 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Nov 2014 09:44:05 -0500 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: <51B09AA7-18DC-4E47-BCC1-DC2D0DD2C126@stufft.io> > On Nov 9, 2014, at 7:13 AM, Vinay Sajip wrote: > >> Thanks, that's very useful feedback. I agree, the need for RDP is very > >> Windows-specific - I don't know how common RDP tools are for Unix, but > > > > > Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access Windows machines via RDP, and it seems fairly stable. There are alternative tools available (such as remmina). > > Regards, > > Vinay Sajip > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig For RDP on OSX, Microsoft has a free RDP app: https://itunes.apple.com/us/app/microsoft-remote-desktop/id715768417 --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sun Nov 9 15:51:42 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 9 Nov 2014 09:51:42 -0500 Subject: [Distutils] pypi.python.org forces SSL? In-Reply-To: <20141108230430.461fe2c1@x230> References: <20141108230430.461fe2c1@x230> Message-ID: > On Nov 8, 2014, at 4:04 PM, Paul Sokolovsky wrote: > > Hello, > > I see that nowadays trying to access http://pypi.python.org (including > any URL beneath it) unconditionally redirects to > https://pypi.python.org. I'm trying to access it using system which > doesn't have SSL support and cannot easily have (embedded, size > constraints). Is there any way to access PyPI metadata/tarballs using > plain HTTP? No, there are no URLs on PyPI which support being accessed by anything other than via TLS. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Sun Nov 9 16:48:32 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 15:48:32 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <51B09AA7-18DC-4E47-BCC1-DC2D0DD2C126@stufft.io> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> <51B09AA7-18DC-4E47-BCC1-DC2D0DD2C126@stufft.io> Message-ID: On 9 November 2014 14:44, Donald Stufft wrote: > For RDP on OSX, Microsoft has a free RDP app: https://itunes.apple.com/us/app/microsoft-remote-desktop/id715768417 Yeah, I don't think RDP is *technically* an issue. There are Linux RDP apps as well. Paul From p.f.moore at gmail.com Sun Nov 9 16:49:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Nov 2014 15:49:45 +0000 Subject: [Distutils] pypi.python.org forces SSL? In-Reply-To: References: <20141108230430.461fe2c1@x230> Message-ID: On 9 November 2014 14:51, Donald Stufft wrote: > No, there are no URLs on PyPI which support being accessed by anything > other than via TLS. You could probably run some sort of local proxy (or something like devpi) that serves pypi content over http. Paul From mail at timgolden.me.uk Sun Nov 9 13:21:43 2014 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 09 Nov 2014 12:21:43 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> Message-ID: <545F5C57.5020406@timgolden.me.uk> On 09/11/2014 12:13, Vinay Sajip wrote: >> Thanks, that's very useful feedback. I agree, the need for RDP is >> very > >> Windows-specific - I don't know how common RDP tools are for Unix, >> but > > > > > Not uncommon, AFAIK. For example, I use rdesktop on Lubuntu to access > Windows machines via RDP, and it seems fairly stable. There are > alternative tools available (such as remmina). I think the OP was speaking not so much about having the technical wherewithal to use RDP but rather about the experience of RDP vs SSH. The difficulty is that Windows doesn't really "think" in ssh. I believe there are (third-party) mechanisms to provide ssh-like access, but I don't know how successful they really are. TJG From kevin.horn at gmail.com Sun Nov 9 19:39:10 2014 From: kevin.horn at gmail.com (Kevin Horn) Date: Sun, 9 Nov 2014 12:39:10 -0600 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> <545F5C57.5020406@timgolden.me.uk> Message-ID: Regarding remote access to windows machines, there are several options: - Remote powershell (not my area, so not sure how viable this is) - Use pexec from sysinternals to run cmd.exe remotely (probably on local network only, and only from other windows machines, so probably not that helpful) - Windows comes with a telnet server (obviously not very secure, but you could use stunnel/vpn or similar to help here) - WinRM (and pywinrm as has been mentioned). Note that vagrant either does or will soon support talking to Windows VMs using this method. - SSH: it is possible to set up an ssh server to work on Windows, but is...non-trivial (i.e. hard) and many caveats apply. See freesshd or KpyM. There are also some GUI options: - RDP - the venerable VNC Obvioiusly the GUI options are more difficult to automate. On Sun, Nov 9, 2014 at 6:28 AM, Paul Moore wrote: > On 9 November 2014 12:21, Tim Golden wrote: > > I think the OP was speaking not so much about having the technical > > wherewithal to use RDP but rather about the experience of RDP vs SSH. > > That was certainly my understanding. The key issue for me is to try to > make the process of just running "pip wheel myproject" or "pip wheel > git+https://github.com/me/myproject" as simple and painless as > possible for people without Windows experience. > > That's somewhat optimistic, because if the command fails with an > error, the developer is still going to need to work out how to debug > why the code isn't portable, etc. But that's a whole different > situation, and well out of scope. > > > The > > difficulty is that Windows doesn't really "think" in ssh. I believe there > > are (third-party) mechanisms to provide ssh-like access, but I don't know > > how successful they really are. > > Yeah, that's where things like cygwin probably won't work well, > because you don't get the "normal" Windows environment. But it might > be possible - after all, see above - it's really only a few simple > commands we need to support. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- -- Kevin Horn -------------- next part -------------- An HTML attachment was scrubbed... URL: From cournape at gmail.com Sun Nov 9 23:35:09 2014 From: cournape at gmail.com (David Cournapeau) Date: Sun, 9 Nov 2014 22:35:09 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> <545F5C57.5020406@timgolden.me.uk> Message-ID: On Sun, Nov 9, 2014 at 6:39 PM, Kevin Horn wrote: > Regarding remote access to windows machines, there are several options: > > - Remote powershell (not my area, so not sure how viable this is) > - Use pexec from sysinternals to run cmd.exe remotely (probably on local > network only, and only from other windows machines, so probably not that > helpful) > - Windows comes with a telnet server (obviously not very secure, but you > could use stunnel/vpn or similar to help here) > - WinRM (and pywinrm as has been mentioned). Note that vagrant either > does or will soon support talking to Windows VMs using this method. > It does (and has for a while). We at Enthought automate package builds for linux, mac and windows through fabric (+ winrm on windows) on vagrant-built VMs, and it works well when you need a consistent interface to those environments. We have been building packages as involved as Qt, Pyside or scipy with native MS tools through this way, so I am confident this would work for almost every situation of interest here. David -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Nov 9 23:54:31 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Nov 2014 11:54:31 +1300 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <51B09AA7-18DC-4E47-BCC1-DC2D0DD2C126@stufft.io> References: <545CF008.1020408@gmail.com> <1415535203.26812.YahooMailNeo@web172405.mail.ir2.yahoo.com> <51B09AA7-18DC-4E47-BCC1-DC2D0DD2C126@stufft.io> Message-ID: <545FF0A7.7050007@canterbury.ac.nz> Donald Stufft wrote: > For RDP on OSX, Microsoft has a free RDP app: > https://itunes.apple.com/us/app/microsoft-remote-desktop/id715768417 And here's another one that I find to be slightly better: http://cord.sourceforge.net/ -- Greg From Steve.Dower at microsoft.com Mon Nov 10 17:35:01 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Mon, 10 Nov 2014 16:35:01 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <85sihu52lg.fsf@benfinney.id.au> References: <851tpf54y5.fsf@benfinney.id.au> <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> <85sihu52lg.fsf@benfinney.id.au> Message-ID: <638dcdc4d36b44ef8d1f68d23c83a875@DM2PR0301MB0734.namprd03.prod.outlook.com> Ben Finney wrote: > Steve Dower writes: >> Ben Finney wrote: >> > The restrictions of the license terms make MS Windows an >> > unacceptable risk on any machine I'm responsible for. >> >> Just out of interest, which restrictions would those be? > > It has been a long time since I bothered to read any of the numerous license > texts from Microsoft, so I can't cite specific clauses. From memory, > unacceptable restrictions include: > > * Restricting the instance to specific hardware, instead of leaving it > up to the recipient to run the work they paid for on any hardware they > choose. If by "specific hardware" you mean the one-license-per-user-per-machine rule, you probably want to consider Windows Server, which has a more flexible license in this respect (or maybe not - it might just allow multiple users on one license/machine. I haven't checked this). > * Forbidding reverse-engineering of the OS to see how it behaves. Yeah, I doubt that restriction is moving anywhere. It's standard for closed-source software, and as I understand it's intended to legally protect trade secrets and patents (i.e. "we tried our hardest to keep this a trade secret"). I've never heard of anyone being pursued for doing it though, except to be offered a job working on Windows :) > * Forbidding collaboration with other recipients to discover how the OS > behaves. "Other recipients" are explicitly excluded - "for use by one person at a time"[1] - so the rest of this point doesn't really make any sense to me. That said, it does trigger some memories of when I was contributing to ReactOS years ago... is this one of their suggestions about how to avoid taint? (Or maybe from Wine?) Those guys have obtained their own legal advice which is going to be aimed at preventing a court case (not just preventing a loss - preventing it from happening in the first place) and so it's going to be based on an interpretation of the license and be more defensive than most people need to worry about. > * Refusal to disclose the source code for the running OS to the > recipient. Again, it's part of the business and legal model. If you really want access to the source code, you can pay for it, but most people and businesses can't afford it or don't want it that badly. (There are also technical reasons why the source code can't easily be disclosed - how many hundreds of gigabytes of code are you willing to download and wade through? Yes, it's that big.) > * Forbidding the recipient from getting their choice of vendor to make > improvements to the OS and collaborate with other recipients on the > improvements. I know this used to exist, as there were a number of RT/embedded OSs available that were based on Windows. I think at this point they've all been absorbed into Microsoft though. > * Arrogating control of the running OS to a party other than the license > recipient, including the ability to (at Microsoft's sole discretion) > deny applications to run, and to disable features of the OS. > > * Arrogating data collection to Microsoft and undisclosed third parties, > tracking broad classes of activity on the OS and sending the logs to a > server not of the recipient's choosing. It seems you fundamentally disagree with the 'licensing' model and would prefer an 'ownership' model. That's fine, but it's not the business model Windows operates under and that is unlikely to ever change. Even if I were CEO, I'd have a hard time changing that one :) >> Does this prevent you from creating a VM on a cloud provider on your >> own account? > > If I need to accept restrictions such as the above, I don't see that the > location of the instance (nor the fees charged) has any affect on these > concerns. The risks discussed above are not mitigated. > >> If the licensing is a real issue, I'm in a position where I can have a >> positive impact on fixing it, so any info you can provide me (on- or >> off-list) about your concerns is valuable. > > Thank you for this offer, I am glad to see willingness expressed to solve these > restrictions. I hope you can achieve software freedom for all recipients of > Microsoft operating systems. > > Until then, the risk is too great to anyone to whom I have professional > responsibilities, and my advice must continue to be that they avoid accepting > such restrictions. That's a fair enough position, and without people taking that stance, Linux (and practically every OS that's based on it) wouldn't be anywhere near as usable as it is today. I'm also fully aware of people with the exact opposite stance who give the exact opposite advice, so there's room in this world for all of us. I'm sorry I can't do any better than the few responses above - these are big issues that run to the core of how Microsoft does business, and not only am I incapable of changing them, I'm nowhere near capable of fully understanding how it all fits together. Thanks for being willing to engage, though. It's always valuable to hear alternative points of view and get a better feeling for how the things we do here are perceived. Rest assured, I will continue to push for as many of our products to be free OSS as I can (we've already put out many developer products under Apache 2.0 or MIT), but it's slow progress and Windows will likely be the last one to make the switch. Cheers, Steve 1: http://download.microsoft.com/Documents/UseTerms/Windows_8_English_ca383862-45cf-467e-97d3-386e0e0260a6.pdf From lists at janc.be Tue Nov 11 02:26:54 2014 From: lists at janc.be (Jan Claeys) Date: Tue, 11 Nov 2014 02:26:54 +0100 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <638dcdc4d36b44ef8d1f68d23c83a875@DM2PR0301MB0734.namprd03.prod.outlook.com> References: <851tpf54y5.fsf@benfinney.id.au> <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> <85sihu52lg.fsf@benfinney.id.au> <638dcdc4d36b44ef8d1f68d23c83a875@DM2PR0301MB0734.namprd03.prod.outlook.com> Message-ID: <1415669214.24622.103.camel@malala> Steve Dower schreef op ma 10-11-2014 om 16:35 [+0000]: > > * Forbidding reverse-engineering of the OS to see how it behaves. > > Yeah, I doubt that restriction is moving anywhere. It's standard for > closed-source software, and as I understand it's intended to legally > protect trade secrets and patents (i.e. "we tried our hardest to keep > this a trade secret"). I've never heard of anyone being pursued for > doing it though, except to be offered a job working on Windows :) FWIW: that statement is illegal and thus void in e.g. the EU (and I thought even the USA?). That's probably why it didn't get pursued often recently... ;) -- Jan Claeys From Steve.Dower at microsoft.com Tue Nov 11 03:57:45 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Tue, 11 Nov 2014 02:57:45 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <1415669214.24622.103.camel@malala> References: <851tpf54y5.fsf@benfinney.id.au> <5d041f9c5799466da8037b3d33be7959@DM2PR0301MB0734.namprd03.prod.outlook.com> <85sihu52lg.fsf@benfinney.id.au> <638dcdc4d36b44ef8d1f68d23c83a875@DM2PR0301MB0734.namprd03.prod.outlook.com>, <1415669214.24622.103.camel@malala> Message-ID: <778c6959e64248579b895e9f055a55e5@DM2PR0301MB0734.namprd03.prod.outlook.com> True. It is followed immediately by a clarification that local laws take precedence, and I guess it's still sufficient to cover the trade secret side of things. Licenses that have to apply internationally are tricky :) Top-posted from my Windows Phone ________________________________ From: Jan Claeys Sent: ?11/?10/?2014 17:33 To: distutils-sig at python.org Subject: Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? Steve Dower schreef op ma 10-11-2014 om 16:35 [+0000]: > > * Forbidding reverse-engineering of the OS to see how it behaves. > > Yeah, I doubt that restriction is moving anywhere. It's standard for > closed-source software, and as I understand it's intended to legally > protect trade secrets and patents (i.e. "we tried our hardest to keep > this a trade secret"). I've never heard of anyone being pursued for > doing it though, except to be offered a job working on Windows :) FWIW: that statement is illegal and thus void in e.g. the EU (and I thought even the USA?). That's probably why it didn't get pursued often recently... ;) -- Jan Claeys _______________________________________________ Distutils-SIG maillist - Distutils-SIG at python.org https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Nov 11 07:27:29 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 11 Nov 2014 06:27:29 +0000 Subject: [Distutils] logs/download numbers on pypi decreased a lot Message-ID: <20141111062729.GB26196@merlinux.eu> Hi Donald, all, i noticed that for several packages daily download numbers are only a tenth or so of what they used to be. This occurs since about a couple of days or a week ago. Any known reason? cheers, holger From donald at stufft.io Tue Nov 11 13:10:22 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 11 Nov 2014 07:10:22 -0500 Subject: [Distutils] logs/download numbers on pypi decreased a lot In-Reply-To: <20141111062729.GB26196@merlinux.eu> References: <20141111062729.GB26196@merlinux.eu> Message-ID: <2BB017C8-A154-4EB4-A844-0553C54C7119@stufft.io> > On Nov 11, 2014, at 1:27 AM, holger krekel wrote: > > Hi Donald, all, > > i noticed that for several packages daily download numbers are only a > tenth or so of what they used to be. This occurs since about a couple > of days or a week ago. Any known reason? > > cheers, > holger As far as I know there have been no changes on our end. Fastly might have changed something. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Tue Nov 11 13:22:25 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 11 Nov 2014 12:22:25 +0000 Subject: [Distutils] logs/download numbers on pypi decreased a lot In-Reply-To: <2BB017C8-A154-4EB4-A844-0553C54C7119@stufft.io> References: <20141111062729.GB26196@merlinux.eu> <2BB017C8-A154-4EB4-A844-0553C54C7119@stufft.io> Message-ID: <20141111122225.GD26196@merlinux.eu> On Tue, Nov 11, 2014 at 07:10 -0500, Donald Stufft wrote: > > Hi Donald, all, > > > > i noticed that for several packages daily download numbers are only a > > tenth or so of what they used to be. This occurs since about a couple > > of days or a week ago. Any known reason? > > > > cheers, > > holger > > As far as I know there have been no changes on our end. Fastly might > have changed something. Any chance to find out from them? If we have such unexplained changes in download numbers the question is if those numbers make any sense at all currently. FWIW you can still see the change in the current data. If you look at e.g. https://pypi.python.org/pypi/matplotlib/ you'll see monthly download of around 135K and daily of ~400 -- if you multiply the latter by 30 you arrive at around 12K -- hence my "only a tenth". holger From setuptools at bugs.python.org Tue Nov 11 17:45:43 2014 From: setuptools at bugs.python.org (Sorin Sbarnea) Date: Tue, 11 Nov 2014 16:45:43 +0000 Subject: [Distutils] [issue159] easy_install --quiet --dry-run -U package is not really quiet Message-ID: <1415724343.7.0.787186080147.issue159@psf.upfronthosting.co.za> New submission from Sorin Sbarnea: easy_install --quiet --dry-run -U packname would still output at least the: install_dir /usr/local/lib/python2.6/dist-packages/ This prevents us from properly running a quite update from crontab, where we are supposed to have output only if there are errors. ---------- messages: 747 nosy: sorin priority: bug status: unread title: easy_install --quiet --dry-run -U package is not really quiet _______________________________________________ Setuptools tracker _______________________________________________ From donald at stufft.io Tue Nov 11 21:13:47 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 11 Nov 2014 15:13:47 -0500 Subject: [Distutils] logs/download numbers on pypi decreased a lot In-Reply-To: <20141111122225.GD26196@merlinux.eu> References: <20141111062729.GB26196@merlinux.eu> <2BB017C8-A154-4EB4-A844-0553C54C7119@stufft.io> <20141111122225.GD26196@merlinux.eu> Message-ID: <202AE4C0-C087-4967-A98A-97D45C54FE55@stufft.io> > On Nov 11, 2014, at 7:22 AM, holger krekel wrote: > > On Tue, Nov 11, 2014 at 07:10 -0500, Donald Stufft wrote: >>> Hi Donald, all, >>> >>> i noticed that for several packages daily download numbers are only a >>> tenth or so of what they used to be. This occurs since about a couple >>> of days or a week ago. Any known reason? >>> >>> cheers, >>> holger >> >> As far as I know there have been no changes on our end. Fastly might >> have changed something. > > Any chance to find out from them? If we have such unexplained changes > in download numbers the question is if those numbers make any sense at > all currently. > > FWIW you can still see the change in the current data. If you look > at e.g. https://pypi.python.org/pypi/matplotlib/ > you'll see monthly download of around 135K and daily of ~400 -- if > you multiply the latter by 30 you arrive at around 12K -- hence my > "only a tenth". > > holger I think I figured it out. A server?s hard disk was filled with temp files because of a bug in software that wasn?t cleaning up on failure. I?ve deployed the fix for that software and cleaned up the temp files. There likely won?t be a backfill of data, we have the raw data but things aren?t set up to easily handle backfills. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Wed Nov 12 11:10:03 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 12 Nov 2014 10:10:03 +0000 Subject: [Distutils] logs/download numbers on pypi decreased a lot In-Reply-To: <202AE4C0-C087-4967-A98A-97D45C54FE55@stufft.io> References: <20141111062729.GB26196@merlinux.eu> <2BB017C8-A154-4EB4-A844-0553C54C7119@stufft.io> <20141111122225.GD26196@merlinux.eu> <202AE4C0-C087-4967-A98A-97D45C54FE55@stufft.io> Message-ID: <20141112101003.GE26196@merlinux.eu> On Tue, Nov 11, 2014 at 15:13 -0500, Donald Stufft wrote: > > On Nov 11, 2014, at 7:22 AM, holger krekel wrote: > > > > On Tue, Nov 11, 2014 at 07:10 -0500, Donald Stufft wrote: > >>> Hi Donald, all, > >>> > >>> i noticed that for several packages daily download numbers are only a > >>> tenth or so of what they used to be. This occurs since about a couple > >>> of days or a week ago. Any known reason? > >>> > >>> cheers, > >>> holger > >> > >> As far as I know there have been no changes on our end. Fastly might > >> have changed something. > > > > Any chance to find out from them? If we have such unexplained changes > > in download numbers the question is if those numbers make any sense at > > all currently. > > > > FWIW you can still see the change in the current data. If you look > > at e.g. https://pypi.python.org/pypi/matplotlib/ > > you'll see monthly download of around 135K and daily of ~400 -- if > > you multiply the latter by 30 you arrive at around 12K -- hence my > > "only a tenth". > > > > holger > > I think I figured it out. A server?s hard disk was filled with temp > files because of a bug in software that wasn?t cleaning up on failure. > I?ve deployed the fix for that software and cleaned up the temp files. > > There likely won?t be a backfill of data, we have the raw data but > things aren?t set up to easily handle backfills. Thanks for caring! a gap of a week is not a big deal IMO. holger > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From robin at reportlab.com Wed Nov 12 12:06:10 2014 From: robin at reportlab.com (Robin Becker) Date: Wed, 12 Nov 2014 11:06:10 +0000 Subject: [Distutils] windows overlay files Message-ID: <54633F22.8040601@chamonix.reportlab.co.uk> Hi, I hope some windows expert can assist me with a production problem. We support a user using windows who reports problems concerning missing attributes. Using GotoMeeting we inspected the file together and see that the attribute should be present. I asked them to zip up the reportlab folder from site-packages and the module that is in the zip does not have the attribute. Puzzlement! Searching reveals this > Due to security features introduced with Windows Vista (UAC) any > non-Administrator program that tries to write to protected locations > such as "Program Files" will get their writes caught and redirected to > an alternative "user friendly" location. > > The program that made the file will be able to see the file, but most > other programs will not. > > Files written to "protected locations" will end up in a parallel file > structure under C:\Users\[username]\AppData\Local\VirtualStore, but > will appear to the program that created them as if actually in the > intended location. so I'm wondering if this is such an issue. The user has a 4 year old version of reportlab and doesn't wish to upgrade. In the past, when they had XP, we used to support minor fixes by modifying the modules and having them overwrite the installed versions in site-packages files. Clearly if some kind of security issues are in place which causes this kind of double file problem, then overwriting the python will not always work. In addition it may be that users can't write the pyc or something. Does anyone here have experience of these issues? Will I be forced to maintain patched installers etc etc? Is there some trick like having an administrator write the files? -- Robin Becker From Steve.Dower at microsoft.com Wed Nov 12 18:49:31 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 12 Nov 2014 17:49:31 +0000 Subject: [Distutils] windows overlay files In-Reply-To: <54633F22.8040601@chamonix.reportlab.co.uk> References: <54633F22.8040601@chamonix.reportlab.co.uk> Message-ID: <82e61fc642064465ae9750629c4c1378@BN1PR0301MB0723.namprd03.prod.outlook.com> Robin Becker wrote: > Hi, > I hope some windows expert can assist me with a production problem. We support a > user using windows who reports problems concerning missing attributes. Using > GotoMeeting we inspected the file together and see that the attribute should be > present. I asked them to zip up the reportlab folder from site-packages and the > module that is in the zip does not have the attribute. Puzzlement! Is this site-packages folder in a normal Python install in x:\Python##? Or has it been installed elsewhere? > Searching reveals this > >> Due to security features introduced with Windows Vista (UAC) any >> non-Administrator program that tries to write to protected locations >> such as "Program Files" will get their writes caught and redirected to >> an alternative "user friendly" location. >> >> The program that made the file will be able to see the file, but most >> other programs will not. >> >> Files written to "protected locations" will end up in a parallel file >> structure under C:\Users\[username]\AppData\Local\VirtualStore, but >> will appear to the program that created them as if actually in the >> intended location. > > > so I'm wondering if this is such an issue. The user has a 4 year old version of > reportlab and doesn't wish to upgrade. In the past, when they had XP, we used to > support minor fixes by modifying the modules and having them overwrite the > installed versions in site-packages files. How does the overwrite work? If they are copying files in Explorer, the redirection you quote above should not occur, but if you have a separate program to do the copying then it will if the program is not run as administrator. > Clearly if some kind of security issues are in place which causes this kind of > double file problem, then overwriting the python will not always work. In > addition it may be that users can't write the pyc or something. The pyc issue should be resolved by generating them at install time. There's an option in the Python installer for this, but any packages added later will need to do it themselves (I believe all of the standard installers do it - pip/easy_install/bdist_win). > Does anyone here have experience of these issues? Will I be forced to maintain > patched installers etc etc? Is there some trick like having an administrator > write the files? If the files need to be written into Program Files, then you'll need to be an administrator - you're making a machine-wide change, which regular users don't have permission to do. That's about all the info I can provide without more details about the customer's file system layout (full paths are really necessary for figuring this out) and the programs/processing being used to install and update these files. Cheers, Steve > -- > Robin Becker > From alex.gaynor at gmail.com Wed Nov 12 20:34:37 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 12 Nov 2014 19:34:37 +0000 (UTC) Subject: [Distutils] Replacing md5 hashes for packages on PyPI Message-ID: Hi everybody! Right now, PyPI provides MD5 hashes for packages, which is used by pip to check for corruption in transit. I'd like to propose we replace MD5 with SHA256 for PyPi, and move to deprecate MD5 support in pip and setuptools. Why should we do this? MD5 is broken. Collision resistance is totally 100% uselessly busted, and pre-image resistance is mathematically broken; practical attacks aren't known publicly, but it's reasonable to assume private attacks are strong because (sing it with me): "Attacks only get better". So MD5 doesn't provide the guarantees one might expect; SHA256 is not broken in these ways. But it's not just not providing value, it's actively causing problems: some machines, such as those with packages compiled to meet FIPS-140-2 do not have MD5 available at all, and so pip's verification raises an exception. While one might be inclined to find a way to silently support both machine configurations, I'd like to instead say we should abhor any additional configuration (whether user supplied or auto-detected) and instead simply upgrade the hashes offered by PyPI, and begin the deprecation process for MD5 in pip. There are currently 60 packages on PyPI which are *not* hosted on PyPI, but do have MD5 hashes there. For these packages we could download the package, verify the MD5 hash, and then upgrade what PyPI stores to be SHA256. Thanks, Alex From donald at stufft.io Wed Nov 12 20:41:13 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 12 Nov 2014 14:41:13 -0500 Subject: [Distutils] Replacing md5 hashes for packages on PyPI In-Reply-To: References: Message-ID: <484D498D-8DE9-440B-BC84-AB4303FB9DC2@stufft.io> > On Nov 12, 2014, at 2:34 PM, Alex Gaynor wrote: > > Hi everybody! > > Right now, PyPI provides MD5 hashes for packages, which is used by pip to > check > for corruption in transit. I'd like to propose we replace MD5 with SHA256 for > PyPi, and move to deprecate MD5 support in pip and setuptools. > > Why should we do this? MD5 is broken. Collision resistance is totally 100% > uselessly busted, and pre-image resistance is mathematically broken; practical > attacks aren't known publicly, but it's reasonable to assume private attacks > are strong because (sing it with me): "Attacks only get better". > > So MD5 doesn't provide the guarantees one might expect; SHA256 is not > broken in > these ways. But it's not just not providing value, it's actively causing > problems: some machines, such as those with packages compiled to meet > FIPS-140-2 do not have MD5 available at all, and so pip's verification raises > an exception. > > While one might be inclined to find a way to silently support both machine > configurations, I'd like to instead say we should abhor any additional > configuration (whether user supplied or auto-detected) and instead simply > upgrade the hashes offered by PyPI, and begin the deprecation process for > MD5 > in pip. > > There are currently 60 packages on PyPI which are *not* hosted on PyPI, but > do > have MD5 hashes there. For these packages we could download the package, > verify > the MD5 hash, and then upgrade what PyPI stores to be SHA256. +1 from me. Security wise pip supported sha256 before it supported TLS so for pip anything that has the ability to securely fetch the sums from the /simple/ pages has the ability to use sha256. For setuptools there was a small window where setuptools implemented TLS verification (0.7) and implemented the support for things other than MD5 (0.9). However I don?t think this small window represents a large (or any?) number of users. IOW the impact should be non-existent other than having better digests. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From graffatcolmingov at gmail.com Thu Nov 13 02:08:10 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Wed, 12 Nov 2014 19:08:10 -0600 Subject: [Distutils] Replacing md5 hashes for packages on PyPI In-Reply-To: <484D498D-8DE9-440B-BC84-AB4303FB9DC2@stufft.io> References: <484D498D-8DE9-440B-BC84-AB4303FB9DC2@stufft.io> Message-ID: +1 from me as well On Wed, Nov 12, 2014 at 1:41 PM, Donald Stufft wrote: > >> On Nov 12, 2014, at 2:34 PM, Alex Gaynor wrote: >> >> Hi everybody! >> >> Right now, PyPI provides MD5 hashes for packages, which is used by pip to >> check >> for corruption in transit. I'd like to propose we replace MD5 with SHA256 for >> PyPi, and move to deprecate MD5 support in pip and setuptools. >> >> Why should we do this? MD5 is broken. Collision resistance is totally 100% >> uselessly busted, and pre-image resistance is mathematically broken; practical >> attacks aren't known publicly, but it's reasonable to assume private attacks >> are strong because (sing it with me): "Attacks only get better". >> >> So MD5 doesn't provide the guarantees one might expect; SHA256 is not >> broken in >> these ways. But it's not just not providing value, it's actively causing >> problems: some machines, such as those with packages compiled to meet >> FIPS-140-2 do not have MD5 available at all, and so pip's verification raises >> an exception. >> >> While one might be inclined to find a way to silently support both machine >> configurations, I'd like to instead say we should abhor any additional >> configuration (whether user supplied or auto-detected) and instead simply >> upgrade the hashes offered by PyPI, and begin the deprecation process for >> MD5 >> in pip. >> >> There are currently 60 packages on PyPI which are *not* hosted on PyPI, but >> do >> have MD5 hashes there. For these packages we could download the package, >> verify >> the MD5 hash, and then upgrade what PyPI stores to be SHA256. > > > +1 from me. > > Security wise pip supported sha256 before it supported TLS so for pip anything > that has the ability to securely fetch the sums from the /simple/ pages has > the ability to use sha256. For setuptools there was a small window where > setuptools implemented TLS verification (0.7) and implemented the support for > things other than MD5 (0.9). However I don?t think this small window represents > a large (or any?) number of users. > > IOW the impact should be non-existent other than having better digests. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From r1chardj0n3s at gmail.com Thu Nov 13 02:29:12 2014 From: r1chardj0n3s at gmail.com (Richard Jones) Date: Thu, 13 Nov 2014 12:29:12 +1100 Subject: [Distutils] Replacing md5 hashes for packages on PyPI In-Reply-To: References: <484D498D-8DE9-440B-BC84-AB4303FB9DC2@stufft.io> Message-ID: +1 pending detail Sent from my mobile - please excuse any brevity. On 13 Nov 2014 12:08, "Ian Cordasco" wrote: > +1 from me as well > > On Wed, Nov 12, 2014 at 1:41 PM, Donald Stufft wrote: > > > >> On Nov 12, 2014, at 2:34 PM, Alex Gaynor wrote: > >> > >> Hi everybody! > >> > >> Right now, PyPI provides MD5 hashes for packages, which is used by pip > to > >> check > >> for corruption in transit. I'd like to propose we replace MD5 with > SHA256 for > >> PyPi, and move to deprecate MD5 support in pip and setuptools. > >> > >> Why should we do this? MD5 is broken. Collision resistance is totally > 100% > >> uselessly busted, and pre-image resistance is mathematically broken; > practical > >> attacks aren't known publicly, but it's reasonable to assume private > attacks > >> are strong because (sing it with me): "Attacks only get better". > >> > >> So MD5 doesn't provide the guarantees one might expect; SHA256 is not > >> broken in > >> these ways. But it's not just not providing value, it's actively causing > >> problems: some machines, such as those with packages compiled to meet > >> FIPS-140-2 do not have MD5 available at all, and so pip's verification > raises > >> an exception. > >> > >> While one might be inclined to find a way to silently support both > machine > >> configurations, I'd like to instead say we should abhor any additional > >> configuration (whether user supplied or auto-detected) and instead > simply > >> upgrade the hashes offered by PyPI, and begin the deprecation process > for > >> MD5 > >> in pip. > >> > >> There are currently 60 packages on PyPI which are *not* hosted on PyPI, > but > >> do > >> have MD5 hashes there. For these packages we could download the package, > >> verify > >> the MD5 hash, and then upgrade what PyPI stores to be SHA256. > > > > > > +1 from me. > > > > Security wise pip supported sha256 before it supported TLS so for pip > anything > > that has the ability to securely fetch the sums from the /simple/ pages > has > > the ability to use sha256. For setuptools there was a small window where > > setuptools implemented TLS verification (0.7) and implemented the > support for > > things other than MD5 (0.9). However I don?t think this small window > represents > > a large (or any?) number of users. > > > > IOW the impact should be non-existent other than having better digests. > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Nov 13 03:31:36 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 12 Nov 2014 21:31:36 -0500 Subject: [Distutils] py2-none-linux_x86_64 tag Message-ID: By popular demand, the development version of wheel now includes the tag for packages that include a binary but don't need a particular Python or ABI - for example, cffi or ctypes packages using dlopen. It's just after the implementation-specific tags. There is not a flag to get bdist_wheel to tag packages this way; you would have to rename them to get wheels bearing the new tag. The complete list on PyPy is currently [('pp27', 'none', 'linux_x86_64'), ('pp27', 'none', 'any'), ('pp2', 'none', 'any'), ('pp26', 'none', 'any'), ('pp25', 'none', 'any'), ('pp24', 'none', 'any'), ('pp23', 'none', 'any'), ('pp22', 'none', 'any'), ('pp21', 'none', 'any'), ('pp20', 'none', 'any'), ('py2', 'none', 'linux_x86_64'), ('py27', 'none', 'any'), ('py2', 'none', 'any'), ('py26', 'none', 'any'), ('py25', 'none', 'any'), ('py24', 'none', 'any'), ('py23', 'none', 'any'), ('py22', 'none', 'any'), ('py21', 'none', 'any'), ('py20', 'none', 'any')] I've also merged a contribution from Benedikt Morbach that sorts the generated metadata to make it deterministic, and have tagged some of the previous released. If there are no complaints I'll probably release as 0.25.0 in a few days. Daniel Holth From fungi at yuggoth.org Thu Nov 13 22:14:40 2014 From: fungi at yuggoth.org (Jeremy Stanley) Date: Thu, 13 Nov 2014 21:14:40 +0000 Subject: [Distutils] py2-none-linux_x86_64 tag In-Reply-To: References: Message-ID: <20141113211439.GM2497@yuggoth.org> On 2014-11-12 21:31:36 -0500 (-0500), Daniel Holth wrote: [...] > If there are no complaints I'll probably release as 0.25.0 in a > few days. I saw the pull request to fix data-only wheels (so that they're not platform-specific) also got merged. Is it safe to assume that will end up in 0.25.0 as well? -- Jeremy Stanley From dholth at gmail.com Thu Nov 13 23:19:28 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 13 Nov 2014 17:19:28 -0500 Subject: [Distutils] py2-none-linux_x86_64 tag In-Reply-To: <20141113211439.GM2497@yuggoth.org> References: <20141113211439.GM2497@yuggoth.org> Message-ID: Yes On Nov 13, 2014 4:42 PM, "Jeremy Stanley" wrote: > On 2014-11-12 21:31:36 -0500 (-0500), Daniel Holth wrote: > [...] > > If there are no complaints I'll probably release as 0.25.0 in a > > few days. > > I saw the pull request to fix data-only wheels (so that they're not > platform-specific) also got merged. Is it safe to assume that will > end up in 0.25.0 as well? > -- > Jeremy Stanley > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Nov 14 03:21:14 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 13 Nov 2014 21:21:14 -0500 Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2 Message-ID: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> Starting a new thread with more explicit details at Richard?s request. Essentially the tl;dr here is that we'll switch to using sha2 (specifically sha256). Simple API ---------- Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the ~60 or so externally hosted files which are using #md5 links will be fetched (one time) verified, and have their #md5= hash replaced with a computed #sha256= hash. Impact: - pip: Will work with no issues, pip has supported sha256 since 1.2, and < 1.2 will install without a hash just fine. - setuptools: Will work with no issues, setuptools has supported sha256 since 0.9 and < 0.9 will install without a hash just fine. - distribute: Doesn't support sha256, will intall without a hash just fine. - buildout: Uses setuptools/distribute to do the downloads I believe. - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out if they do not exist. JSON / XMLRPC API ----------------- Keep the md5_sum field, add an additional sha256_sum, suggest that applications switch to using sha256 for verification. Impact: - bandersnatch: bandersnatch will continue to use the md5_sum field from the JSON (and previously XMLRPC) and should be updated to using sha256 in the future. Web UI ------ Simply replace any use of MD5 with SHA256, no clients are expected to access anything here so this should be perfectly fine. Other Clients ------------- - pep381client: Doesn't do anything special with the hash, will continue to work. - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I can't determine where it gets the md5sum from and what it will do if it doesn't exist. It does have some handling of md5 though. List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, which is generated by looking at what is downloading the files from PyPI. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From richard at python.org Fri Nov 14 05:48:52 2014 From: richard at python.org (Richard Jones) Date: Fri, 14 Nov 2014 15:48:52 +1100 Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2 In-Reply-To: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> Message-ID: +1 thanks for the detail On 14 November 2014 13:21, Donald Stufft wrote: > Starting a new thread with more explicit details at Richard?s request. > Essentially the tl;dr here is that we'll switch to using sha2 (specifically > sha256). > > > Simple API > ---------- > > Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, > the > ~60 or so externally hosted files which are using #md5 links will be > fetched > (one time) verified, and have their #md5= hash replaced with a computed > #sha256= hash. > > Impact: > - pip: Will work with no issues, pip has supported sha256 since 1.2, and > < 1.2 will install without a hash just fine. > - setuptools: Will work with no issues, setuptools has supported sha256 > since > 0.9 and < 0.9 will install without a hash just fine. > - distribute: Doesn't support sha256, will intall without a hash just > fine. > - buildout: Uses setuptools/distribute to do the downloads I believe. > - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error > out > if they do not exist. > > > JSON / XMLRPC API > ----------------- > > Keep the md5_sum field, add an additional sha256_sum, suggest that > applications > switch to using sha256 for verification. > > Impact: > - bandersnatch: bandersnatch will continue to use the md5_sum field from > the > JSON (and previously XMLRPC) and should be updated to > using > sha256 in the future. > > > Web UI > ------ > > Simply replace any use of MD5 with SHA256, no clients are expected to > access > anything here so this should be perfectly fine. > > > Other Clients > ------------- > > - pep381client: Doesn't do anything special with the hash, will continue > to > work. > - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I > can't determine where it gets the md5sum from and what it will > do if > it doesn't exist. It does have some handling of md5 though. > > > List of clients to look at taken from > http://d.stufft.io/image/402r1s442m2r, > which is generated by looking at what is downloading the files from PyPI. > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Fri Nov 14 07:38:09 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 14 Nov 2014 06:38:09 +0000 Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2 In-Reply-To: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> Message-ID: <20141114063809.GV26196@merlinux.eu> Hi Donald, thanks for the detail and the pre-announcement! I am all for the change but indeed need to check how devpi code is affected (pretty sure it is) and how to accomodate the change. Will see to do so next week and get back to this thread. best, holger On Thu, Nov 13, 2014 at 21:21 -0500, Donald Stufft wrote: > Starting a new thread with more explicit details at Richard?s request. > Essentially the tl;dr here is that we'll switch to using sha2 (specifically > sha256). > > > Simple API > ---------- > > Drop the #md5= from the PyPI hosted tarballs and replace it with #sha256, the > ~60 or so externally hosted files which are using #md5 links will be fetched > (one time) verified, and have their #md5= hash replaced with a computed > #sha256= hash. > > Impact: > - pip: Will work with no issues, pip has supported sha256 since 1.2, and > < 1.2 will install without a hash just fine. > - setuptools: Will work with no issues, setuptools has supported sha256 since > 0.9 and < 0.9 will install without a hash just fine. > - distribute: Doesn't support sha256, will intall without a hash just fine. > - buildout: Uses setuptools/distribute to do the downloads I believe. > - z3c.pypimirror: Appears to use MD5 hashes, but appears it won't error out > if they do not exist. > > > JSON / XMLRPC API > ----------------- > > Keep the md5_sum field, add an additional sha256_sum, suggest that applications > switch to using sha256 for verification. > > Impact: > - bandersnatch: bandersnatch will continue to use the md5_sum field from the > JSON (and previously XMLRPC) and should be updated to using > sha256 in the future. > > > Web UI > ------ > > Simply replace any use of MD5 with SHA256, no clients are expected to access > anything here so this should be perfectly fine. > > > Other Clients > ------------- > > - pep381client: Doesn't do anything special with the hash, will continue to > work. > - devpi: ??? Unsure, I don't follow the code which fetches from PyPI so I > can't determine where it gets the md5sum from and what it will do if > it doesn't exist. It does have some handling of md5 though. > > > List of clients to look at taken from http://d.stufft.io/image/402r1s442m2r, > which is generated by looking at what is downloading the files from PyPI. > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From p.f.moore at gmail.com Sat Nov 15 11:45:05 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 15 Nov 2014 10:45:05 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: On 7 November 2014 15:46, Paul Moore wrote: > To that end, I'd like to get an idea of what sort of access to Windows > a typical Unix developer would have. Thanks to all who contributed to this thread. Based on the feedback, I think it's going to be useful to provide two options. First of all, an EC2 AMI that can be used by people without access to a local Windows system. While other cloud providers are a possibility, EC2 provides a free tier (for the first year) and is well-known, so it's probably the easiest to get started with (at least it was for me!) Also, I will provide a script that can be used to automatically build the environment on a newly-installed machine. The idea is that you can use this on a Windows VM (something that a number of people have said they have access to). The script may be usable on an existing machine, but it's hard to make it robust, as there are too many failure modes to consider (software already installed, configuration and/or permission differences, etc). So while such use may be possible, I probably won't consider it as supported. Thanks again, Paul From shimizukawa at gmail.com Wed Nov 19 16:16:21 2014 From: shimizukawa at gmail.com (Takayuki Shimizukawa) Date: Thu, 20 Nov 2014 00:16:21 +0900 Subject: [Distutils] Requesting PyPI classifiers for Sphinx Message-ID: Hi, I'd like to request classifiers for the Sphinx like: "Topic :: Documentation :: Sphinx" "Topic :: Documentation :: Sphinx :: Extension" It would help us to search sphinx extensions and sphinx related tools. .. ex. sphinx-intl, sphinx-testing, Tinkerer, and sphinx extensions. .. refs: "[sphinx-users]: Survey of Sphinx extensions" [1]_ [2]_ Thanks. .. [1] https://groups.google.com/d/msg/sphinx-users/p82dKvqCUck/BBoavGgXRcwJ .. [2] http://sphinxext-survey.readthedocs.org/en/latest/ -- Takayuki SHIMIZUKAWA http://about.me/shimizukawa From holger at merlinux.eu Thu Nov 20 14:31:35 2014 From: holger at merlinux.eu (holger krekel) Date: Thu, 20 Nov 2014 13:31:35 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: References: Message-ID: <20141120133135.GD25600@merlinux.eu> On Sat, Nov 15, 2014 at 10:45 +0000, Paul Moore wrote: > On 7 November 2014 15:46, Paul Moore wrote: > > To that end, I'd like to get an idea of what sort of access to Windows > > a typical Unix developer would have. > > Thanks to all who contributed to this thread. > > Based on the feedback, I think it's going to be useful to provide two > options. First of all, an EC2 AMI that can be used by people without > access to a local Windows system. While other cloud providers are a > possibility, EC2 provides a free tier (for the first year) and is > well-known, so it's probably the easiest to get started with (at least > it was for me!) Also, I will provide a script that can be used to > automatically build the environment on a newly-installed machine. The > idea is that you can use this on a Windows VM (something that a number > of people have said they have access to). > > The script may be usable on an existing machine, but it's hard to make > it robust, as there are too many failure modes to consider (software > already installed, configuration and/or permission differences, etc). > So while such use may be possible, I probably won't consider it as > supported. Thanks Paul for going through this! Looking forward to the link/code. holger > Thanks again, > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From p.f.moore at gmail.com Thu Nov 20 22:51:18 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 20 Nov 2014 21:51:18 +0000 Subject: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows? In-Reply-To: <20141120133135.GD25600@merlinux.eu> References: <20141120133135.GD25600@merlinux.eu> Message-ID: On 20 November 2014 13:31, holger krekel wrote: > Thanks Paul for going through this! Looking forward to the link/code. Cheers - it's not forgotten, but real life's being a nuisance, so it's on the back burner for a short while. I'll try to get something done in a few weeks. Paul From donquestion at rocketmail.com Sat Nov 22 16:49:56 2014 From: donquestion at rocketmail.com (Donaldo Fastoso) Date: Sat, 22 Nov 2014 16:49:56 +0100 Subject: [Distutils] setup.cfg Message-ID: <5470B0A4.6000604@rocketmail.com> Hi, is there a way for python 2.7 to write something like: setup.py: from distutils.core import setup setup() setup.cfg: [metadata] name=foo version=0.1 and get on the cmdline: python setup.py --name foo in distutils2 this was possible. Has python 2.7 stdlib distutils similar capabilities? May i use the [global] section? (as for now i only get errors trying that.) Or should i use the [metadata] command to implement it myself? From sontek at gmail.com Sun Nov 23 02:24:36 2014 From: sontek at gmail.com (John Anderson) Date: Sat, 22 Nov 2014 17:24:36 -0800 Subject: [Distutils] Managing how an egg-info is created? Message-ID: We use gunicorn as our webserver and we use it to do zero downtime deployments. The way it does this is by having a master process that forks children. This works perfectly when we deploy as editable installations (pip install -e) but doesn't work when we install sdists. The reason this doesn't work with sdists is because the .egg-info directory is named by the version, for example, if we deploy AnWeb-1.0 and it'll give a path like this: site-packages /AnWeb-1.0.egg-info/ Then we release AnWeb-1.5, it'll give us: site-packages/AnWeb-1.5.egg-info/ So when a new worker is forked, the master already has sys.path loaded and its going to check for sites-packages/AnWeb-1.0.egg-info/entry_points.txt which will fail and the workers die. What I'm wondering is if I can control this somehow to get a non-versioned egg-info installed from an sdist? Or do you recommend always using editable installs? -------------- next part -------------- An HTML attachment was scrubbed... URL: From shimizukawa at gmail.com Tue Nov 25 02:07:03 2014 From: shimizukawa at gmail.com (Takayuki Shimizukawa) Date: Tue, 25 Nov 2014 01:07:03 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: Message-ID: Bump. I'd like to request trove classifiers for the Sphinx like: "Topic :: Documentation :: Sphinx" "Topic :: Documentation :: Sphinx :: Extension" And additionally: "Topic :: Documentation :: Sphinx :: Theme" Thanks. On Thu Nov 20 2014 at 0:16:21 Takayuki Shimizukawa wrote: > Hi, > > I'd like to request classifiers for the Sphinx like: > > "Topic :: Documentation :: Sphinx" > "Topic :: Documentation :: Sphinx :: Extension" > > It would help us to search sphinx extensions and sphinx related tools. > .. ex. sphinx-intl, sphinx-testing, Tinkerer, and sphinx extensions. > .. refs: "[sphinx-users]: Survey of Sphinx extensions" [1]_ [2]_ > > Thanks. > > .. [1] https://groups.google.com/d/msg/sphinx-users/p82dKvqCUck/ > BBoavGgXRcwJ > .. [2] http://sphinxext-survey.readthedocs.org/en/latest/ > -- > Takayuki SHIMIZUKAWA > http://about.me/shimizukawa > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pydanny at gmail.com Tue Nov 25 06:48:42 2014 From: pydanny at gmail.com (Daniel Greenfeld) Date: Mon, 24 Nov 2014 21:48:42 -0800 Subject: [Distutils] Requesting PyPI classifiers for Sphinx In-Reply-To: References: Message-ID: +1 --Danny On Nov 24, 2014 5:07 PM, "Takayuki Shimizukawa" wrote: > Bump. > > I'd like to request trove classifiers for the Sphinx like: > > "Topic :: Documentation :: Sphinx" > "Topic :: Documentation :: Sphinx :: Extension" > > And additionally: > > "Topic :: Documentation :: Sphinx :: Theme" > > Thanks. > > On Thu Nov 20 2014 at 0:16:21 Takayuki Shimizukawa > wrote: > >> Hi, >> >> I'd like to request classifiers for the Sphinx like: >> >> "Topic :: Documentation :: Sphinx" >> "Topic :: Documentation :: Sphinx :: Extension" >> >> It would help us to search sphinx extensions and sphinx related tools. >> .. ex. sphinx-intl, sphinx-testing, Tinkerer, and sphinx extensions. >> .. refs: "[sphinx-users]: Survey of Sphinx extensions" [1]_ [2]_ >> >> Thanks. >> >> .. [1] https://groups.google.com/d/msg/sphinx-users/p82dKvqCUck/ >> BBoavGgXRcwJ >> .. [2] http://sphinxext-survey.readthedocs.org/en/latest/ >> -- >> Takayuki SHIMIZUKAWA >> http://about.me/shimizukawa >> > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Tue Nov 25 15:11:53 2014 From: fred at fdrake.net (Fred Drake) Date: Tue, 25 Nov 2014 09:11:53 -0500 Subject: [Distutils] Requesting PyPI classifiers for Sphinx In-Reply-To: References: Message-ID: On Wed, Nov 19, 2014 at 10:16 AM, Takayuki Shimizukawa wrote: > I'd like to request classifiers for the Sphinx like: > > "Topic :: Documentation :: Sphinx" > "Topic :: Documentation :: Sphinx :: Extension" I suspect adding "Framework :: Sphinx" would make more sense for extensions. For tools used with Sphinx documentation without being tied to the Sphinx application itself, "Topic :: Documentation :: Sphinx" seems fine. (I'd imagine an extension would specify both, and non-extension tools would use only "Topic :: Documentation :: Sphinx".) -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From erik.m.bray at gmail.com Wed Nov 26 00:35:53 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 25 Nov 2014 18:35:53 -0500 Subject: [Distutils] setup.cfg In-Reply-To: <5470B0A4.6000604@rocketmail.com> References: <5470B0A4.6000604@rocketmail.com> Message-ID: On Sat, Nov 22, 2014 at 10:49 AM, Donaldo Fastoso wrote: > Hi, > > is there a way for python 2.7 to write something like: > > setup.py: > > from distutils.core import setup > setup() > > setup.cfg: > > [metadata] > name=foo > version=0.1 > > > and get on the cmdline: > > python setup.py --name > foo > > > in distutils2 this was possible. Has python 2.7 stdlib distutils similar > capabilities? May i use the [global] section? (as for now i only get > errors trying that.) Or should i use the [metadata] command to implement > it myself? You might consider giving d2to1 a whirl, if it suits your needs otherwise. It's long been on my todo list to release a successor to d2to1 that is better documented and not as tied to the ill-fated distutils2 project. But in the meantime d2to1 does work and pretty well at that. Unfortunately other priorities have kept updates to it on the backburner. You might also consider giving pbr a look: http://docs.openstack.org/developer/pbr/ This is basically a fork of d2to1 made for OpenStack that I think may be more actively developed. Erik From ncoghlan at gmail.com Wed Nov 26 07:44:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Nov 2014 16:44:32 +1000 Subject: [Distutils] setup.cfg In-Reply-To: References: <5470B0A4.6000604@rocketmail.com> Message-ID: On 26 November 2014 at 09:35, Erik Bray wrote: > You might also consider giving pbr a look: > > http://docs.openstack.org/developer/pbr/ > > This is basically a fork of d2to1 made for OpenStack that I think may > be more actively developed. No "may" about it - the OpenStack folks continue to iterate on PBR to ensure it meets the needs of the various OpenStack projects. Robert Collins gave a talk about semantic versioning with PBR at PyCon New Zealand: https://www.youtube.com/watch?v=6hjg-Uia5AU It's most effective if you're using git for version control, but some aspects should be useful regardless of the specific version control system. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From shimizukawa at gmail.com Wed Nov 26 11:46:44 2014 From: shimizukawa at gmail.com (Takayuki Shimizukawa) Date: Wed, 26 Nov 2014 10:46:44 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: Message-ID: Hi Fred, On Tue Nov 25 2014 at 23:12:14 Fred Drake wrote: > On Wed, Nov 19, 2014 at 10:16 AM, Takayuki Shimizukawa > wrote: > > I'd like to request classifiers for the Sphinx like: > > > > "Topic :: Documentation :: Sphinx" > > "Topic :: Documentation :: Sphinx :: Extension" > > I suspect adding "Framework :: Sphinx" would make more sense for > extensions. For tools used with Sphinx documentation without being > tied to the Sphinx application itself, "Topic :: Documentation :: > Sphinx" seems fine. (I'd imagine an extension would specify both, and > non-extension tools would use only "Topic :: Documentation :: > Sphinx".) > Thanks for the suggestion. It make sense to me too. How about this? "Topic :: Documentation :: Sphinx" "Framework :: Sphinx" "Framework :: Sphinx :: Extension" "Framework :: Sphinx :: Theme" Regards, -- Takayuki Shimizukawa http://about.me/shimizukawa -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Wed Nov 26 12:25:37 2014 From: fred at fdrake.net (Fred Drake) Date: Wed, 26 Nov 2014 06:25:37 -0500 Subject: [Distutils] Requesting PyPI classifiers for Sphinx In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 5:46 AM, Takayuki Shimizukawa wrote: > How about this? > > "Topic :: Documentation :: Sphinx" > "Framework :: Sphinx" > "Framework :: Sphinx :: Extension" > "Framework :: Sphinx :: Theme" +1 -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From donquestion at rocketmail.com Wed Nov 26 21:40:38 2014 From: donquestion at rocketmail.com (Donaldo Fastoso) Date: Wed, 26 Nov 2014 21:40:38 +0100 Subject: [Distutils] Requesting PyPI classifiers for Sphinx In-Reply-To: References: Message-ID: <54763AC6.7090207@rocketmail.com> Somewhat OT, but i was wondering if there is documentation helping to choose the right classifiers if the choice is not obvious. For example i find the "Development Status" classifiers somewhat "quirky": 3 classifiers for alpha-releases: Development Status :: 1 - Planning Development Status :: 2 - Pre-Alpha Development Status :: 3 - Alpha but no "release candidate" Development Status :: 4 - Beta Development Status :: 5 - Production/Stable and similar to the alphas a somewhat murky distinction between the stables: Development Status :: 5 - Production/Stable Development Status :: 6 - Mature What is the rationale behind this and where can i inform myself about it? And on a second thought: is there a way to deprecate classifiers if they get obsolete? From richard at python.org Wed Nov 26 23:18:35 2014 From: richard at python.org (Richard Jones) Date: Wed, 26 Nov 2014 22:18:35 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: <54763AC6.7090207@rocketmail.com> Message-ID: On Thu Nov 27 2014 at 9:08:40 AM Donaldo Fastoso wrote: > Somewhat OT, > > but i was wondering if there is documentation helping to choose > the right classifiers if the choice is not obvious. > > For example i find the "Development Status" classifiers somewhat > "quirky": 3 classifiers for alpha-releases: > > Development Status :: 1 - Planning > Development Status :: 2 - Pre-Alpha > Development Status :: 3 - Alpha > > but no "release candidate" > I'm not sure that's a level of granularity that the cheese shop really needs. Release candidates can be tagged "beta". > > Development Status :: 4 - Beta > Development Status :: 5 - Production/Stable > > and similar to the alphas a somewhat murky distinction between the > stables: > > Development Status :: 5 - Production/Stable > Development Status :: 6 - Mature > That is a somewhat murky distinction, yes. In my mind mature applies to packages that have been stable for a long time and have not had any significant work done on them. Something like gadfly, for example :) > And on a second thought: is there a way to deprecate classifiers if > they get obsolete? > No, they're there for liff :) Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From fred at fdrake.net Wed Nov 26 23:25:54 2014 From: fred at fdrake.net (Fred Drake) Date: Wed, 26 Nov 2014 17:25:54 -0500 Subject: [Distutils] Requesting PyPI classifiers for Sphinx In-Reply-To: References: Message-ID: On Wed, Nov 26, 2014 at 5:15 PM, Richard Jones wrote: > Do you have any idea of the number of existing packages that would fit under > each of those categories? I'm only asking because I wonder whether we need > all four right now. A quick search using the PyPI search box suggests there are a *lot* of extensions (a couple of hundred?) and even themes (dozens; more than I expected). I'm not sure how to search for sphinx-related software other than extensions or themes with that UI. I was surprised at just how many relevant packages were listed (a testament to Sphinx's well-deserved success); I suspect the categories are justified. Getting the appropriate package maintainers to be aware of and apply them is another matter. That gets back to the guidance question, which is really separate from this. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From richard at python.org Wed Nov 26 23:15:07 2014 From: richard at python.org (Richard Jones) Date: Wed, 26 Nov 2014 22:15:07 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: Message-ID: On Wed Nov 26 2014 at 9:47:07 PM Takayuki Shimizukawa wrote: > Hi Fred, > > On Tue Nov 25 2014 at 23:12:14 Fred Drake wrote: > >> On Wed, Nov 19, 2014 at 10:16 AM, Takayuki Shimizukawa >> wrote: >> > I'd like to request classifiers for the Sphinx like: >> > >> > "Topic :: Documentation :: Sphinx" >> > "Topic :: Documentation :: Sphinx :: Extension" >> >> I suspect adding "Framework :: Sphinx" would make more sense for >> extensions. For tools used with Sphinx documentation without being >> tied to the Sphinx application itself, "Topic :: Documentation :: >> Sphinx" seems fine. (I'd imagine an extension would specify both, and >> non-extension tools would use only "Topic :: Documentation :: >> Sphinx".) >> > > Thanks for the suggestion. It make sense to me too. > How about this? > > "Topic :: Documentation :: Sphinx" > "Framework :: Sphinx" > "Framework :: Sphinx :: Extension" > "Framework :: Sphinx :: Theme" > Do you have any idea of the number of existing packages that would fit under each of those categories? I'm only asking because I wonder whether we need all four right now. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Nov 26 23:32:13 2014 From: richard at python.org (Richard Jones) Date: Wed, 26 Nov 2014 22:32:13 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: Message-ID: Thanks for that, Fred. I've added all four classifiers. Richard On Thu Nov 27 2014 at 9:26:24 AM Fred Drake wrote: > On Wed, Nov 26, 2014 at 5:15 PM, Richard Jones wrote: > > Do you have any idea of the number of existing packages that would fit > under > > each of those categories? I'm only asking because I wonder whether we > need > > all four right now. > > A quick search using the PyPI search box suggests there are a *lot* of > extensions (a couple of hundred?) and even themes (dozens; more than I > expected). I'm not sure how to search for sphinx-related software > other than extensions or themes with that UI. > > I was surprised at just how many relevant packages were listed (a > testament to Sphinx's well-deserved success); I suspect the categories > are justified. Getting the appropriate package maintainers to be > aware of and apply them is another matter. That gets back to the > guidance question, which is really separate from this. > > > -Fred > > -- > Fred L. Drake, Jr. > "A storm broke loose in my mind." --Albert Einstein > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shimizukawa at gmail.com Thu Nov 27 01:53:48 2014 From: shimizukawa at gmail.com (Takayuki Shimizukawa) Date: Thu, 27 Nov 2014 00:53:48 +0000 Subject: [Distutils] Requesting PyPI classifiers for Sphinx References: Message-ID: Thanks Fred, Richard. I'll announce it to sphinx-users and developers. I'll also change sphinx-contrib package template https://bitbucket.org/birkenfeld/sphinx-contrib/src/tip/_template/ and update several tutorials. Thanks, On Thu Nov 27 2014 at 7:32:13 Richard Jones wrote: > On Thu Nov 27 2014 at 9:26:24 AM Fred Drake wrote: > >> On Wed, Nov 26, 2014 at 5:15 PM, Richard Jones >> wrote: >> > Do you have any idea of the number of existing packages that would fit >> under >> > each of those categories? I'm only asking because I wonder whether we >> need >> > all four right now. >> >> A quick search using the PyPI search box suggests there are a *lot* of >> extensions (a couple of hundred?) and even themes (dozens; more than I >> expected). I'm not sure how to search for sphinx-related software >> other than extensions or themes with that UI. >> >> I was surprised at just how many relevant packages were listed (a >> testament to Sphinx's well-deserved success); I suspect the categories >> are justified. Getting the appropriate package maintainers to be >> aware of and apply them is another matter. That gets back to the >> guidance question, which is really separate from this. >> >> >> -Fred >> >> -- >> Fred L. Drake, Jr. >> "A storm broke loose in my mind." --Albert Einstein >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From randy.syring at level12.io Thu Nov 27 18:47:05 2014 From: randy.syring at level12.io (Randy Syring) Date: Thu, 27 Nov 2014 12:47:05 -0500 Subject: [Distutils] What binary formats should we be distributing? Message-ID: <54776399.708@level12.io> pymssql uses Cython and the FreeTDS C library to support connections to Microsoft SQL Server. We would like to distribute binary packages to ease installation for users. Our current build matrix looks like this: https://github.com/pymssql/pymssql/wiki/Build-Matrix However, we are trying to cut that down some since, by the time we support 5 versions of python (2.6,2.7,3.2,3.3,3.4), we are having to release 30 and now, since we are considering .msi files, maybe 40 files per release. I'm trying to figure out if there are any downsides to: * replacing our .exe with .msi (lxml uses .exe, but we noticed Python itself is shipping .msi files now) * only shipping wheels and no longer shipping eggs I would also appreciate any other thoughts regarding what kind of binary formats a "responsible" package owner would ship. Thanks. *Randy Syring* Chief Executive Developer Direct: 502.276.0459 Office: 812.285.8766 Level 12 Technologies /Principled People | Technology that Works/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Level-12-Logo-email-sigs.png Type: image/png Size: 4071 bytes Desc: not available URL: From p.f.moore at gmail.com Fri Nov 28 00:25:41 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 27 Nov 2014 23:25:41 +0000 Subject: [Distutils] What binary formats should we be distributing? In-Reply-To: <54776399.708@level12.io> References: <54776399.708@level12.io> Message-ID: On 27 November 2014 at 17:47, Randy Syring wrote: > pymssql uses Cython and the FreeTDS C library to support connections to > Microsoft SQL Server. We would like to distribute binary packages to ease > installation for users. Our current build matrix looks like this: > > https://github.com/pymssql/pymssql/wiki/Build-Matrix > > However, we are trying to cut that down some since, by the time we support > 5 versions of python (2.6,2.7,3.2,3.3,3.4), we are having to release 30 and > now, since we are considering .msi files, maybe 40 files per release. > > I'm trying to figure out if there are any downsides to: > > - replacing our .exe with .msi (lxml uses .exe, but we noticed Python > itself is shipping .msi files now) > - only shipping wheels and no longer shipping eggs > > I would also appreciate any other thoughts regarding what kind of binary > formats a "responsible" package owner would ship. > I can only really comment on Windows, but I would say that wheel is sufficient there. The longer term intent is that wheel supersedes all of the other binary formats as a universal standard. There's a way to go yet, so there may be reasons to still ship other formats, but you should be looking to phase them out once your users no longer need them. I would strongly recommend against msi, as they are not typically useful for Python packages. Sure, the Python distribution itself uses msi, but the reasons for that are not really applicable to Python packages. The bdist_msi format is very rarely used in practice, and that is likely to decrease rather than increase. The one proviso that I would make is that this is from a hobbyist/open source perspective. I'd guess from the fact that you're distributing a SQL Server library that you have a larger than average proportion of "corporate" users. In that sort of environment, MSI installers may be more useful, as they integrate better with corporate deployment tools. But that's not something I can really comment on. The wininst (exe) format is probably of limited value, It is of use for systems without pip where integration with add/remove programs is useful, but such systems are probably rare. But if you want to distribute a "standalone installer" format, exe is probably more familiar to Python package users than msi. On the downside, it doesn't work with virtualenvs. As regards eggs, they are essentially the precursor to wheels as a binary distribution format. Users still using easy_install will need eggs (but why aren't they using pip these days?). It's possible that older systems such as buildout may still need eggs, but I'm not familiar with these systems and don't know if they can use wheels instead these days. I hope this helps. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Nov 28 07:03:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 28 Nov 2014 16:03:59 +1000 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files Message-ID: We've discussed the idea of changing the wheel file naming scheme to deal with Linux previously, but never put together a concrete proposal. The closest we've got is the idea of allowing the platform tag to be customised in pip and perhaps bdist_wheel, and while that's good from an "enabling experimentation" perspective, it may be overkill if the primary goal is just to better support handling of Linux distros. For starters, here's the current definition of the platform tag in PEP 425: ================= The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _ . * win32 * linux_i386 * linux_x86_64 ================= Here's my proposed change: ================= The default platform tag is distutils.util.get_platform() with all hyphens - and periods . replaced with underscore _ . If /etc/os-release [N] exists on the system, then the values in the 'ID' and 'VERSION_ID' fields are read, all hyphens - and periods . replaced with underscore _ , and the results appended to the default tag after a separating underscore." Examples: * win32 * macosx_10_6_intel * linux_x86_64_fedora_20 * linux_x86_64_rhel_7_0 * linux_x86_64_debian_7_0 * linux_x86_64_ubuntu_14_04 ================= The [N] reference would then be a reference to http://www.freedesktop.org/software/systemd/man/os-release.html for the definition of the format of os-release. (Note that while the format originated with systemd, plenty of distros have also started providing it regardless of which init system they use) Now, this slightly overspecifies on the *consumer* side. A binary wheel that works on "rhel_7_0" for example, should almost certainly work on "rhel_7_1". However, that can be addressed on the tooling side (e.g. permitting the specification of "additional compatible platforms" when invoking pip), rather than needing to be in the specification. This also won't help with older Linux distros that don't offer /etc/os-release, but I'm OK with that - those can just continue to show up as "linux_x86_64", and PyPI can continue to disallow those uploads. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Fri Nov 28 09:19:51 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Nov 2014 09:19:51 +0100 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files References: Message-ID: <20141128091951.0ab5b751@fsol> On Fri, 28 Nov 2014 16:03:59 +1000 Nick Coghlan wrote: > Here's my proposed change: > > ================= > The default platform tag is distutils.util.get_platform() with all > hyphens - and periods . replaced with underscore _ . If > /etc/os-release [N] exists on the system, then the values in the 'ID' > and 'VERSION_ID' fields are read, all hyphens - and periods . replaced > with underscore _ , and the results appended to the default tag after > a separating underscore." > > Examples: > > * win32 > * macosx_10_6_intel > * linux_x86_64_fedora_20 > * linux_x86_64_rhel_7_0 > * linux_x86_64_debian_7_0 > * linux_x86_64_ubuntu_14_04 Is this not going to be a slippery slope? > Now, this slightly overspecifies on the *consumer* side. A binary > wheel that works on "rhel_7_0" for example, should almost certainly > work on "rhel_7_1". However, that can be addressed on the tooling side > (e.g. permitting the specification of "additional compatible > platforms" when invoking pip), rather than needing to be in the > specification. How about those lesser known distributions (e.g. Linux Mint or Mageia)? How many binary packages will package authors have to provide to cover people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied by three or four Python versions is already a lot of binaries to build... While this would be a good technical solution, I think it's socially disastrous. Of course, you may point out that it has its roots in the failure of the GNU/Linux ecosystem to provide real binary compatibility. It's stunning that under Windows you can build a Windows XP-compatible shared library with a recent MSVC just by turning a switch in the options... Regards Antoine. From marius at pov.lt Fri Nov 28 09:36:45 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 28 Nov 2014 10:36:45 +0200 Subject: [Distutils] What binary formats should we be distributing? In-Reply-To: References: <54776399.708@level12.io> Message-ID: <20141128083645.GA3505@fridge.pov.lt> On Thu, Nov 27, 2014 at 11:25:41PM +0000, Paul Moore wrote: > On 27 November 2014 at 17:47, Randy Syring wrote: > > > pymssql uses Cython and the FreeTDS C library to support connections to > > Microsoft SQL Server. We would like to distribute binary packages to ease > > installation for users. Our current build matrix looks like this: > > > > https://github.com/pymssql/pymssql/wiki/Build-Matrix > > > > However, we are trying to cut that down some since, by the time we support > > 5 versions of python (2.6,2.7,3.2,3.3,3.4), we are having to release 30 and > > now, since we are considering .msi files, maybe 40 files per release. > > > > I'm trying to figure out if there are any downsides to: > > > > - replacing our .exe with .msi (lxml uses .exe, but we noticed Python > > itself is shipping .msi files now) > > - only shipping wheels and no longer shipping eggs > > > > I would also appreciate any other thoughts regarding what kind of binary > > formats a "responsible" package owner would ship. > > I can only really comment on Windows, but I would say that wheel is > sufficient there. The longer term intent is that wheel supersedes all of > the other binary formats as a universal standard. There's a way to go yet, > so there may be reasons to still ship other formats, but you should be > looking to phase them out once your users no longer need them. > > I would strongly recommend against msi, as they are not typically useful > for Python packages. Sure, the Python distribution itself uses msi, but the > reasons for that are not really applicable to Python packages. The > bdist_msi format is very rarely used in practice, and that is likely to > decrease rather than increase. The one proviso that I would make is that > this is from a hobbyist/open source perspective. I'd guess from the fact > that you're distributing a SQL Server library that you have a larger than > average proportion of "corporate" users. In that sort of environment, MSI > installers may be more useful, as they integrate better with corporate > deployment tools. But that's not something I can really comment on. > > The wininst (exe) format is probably of limited value, It is of use for > systems without pip where integration with add/remove programs is useful, > but such systems are probably rare. But if you want to distribute a > "standalone installer" format, exe is probably more familiar to Python > package users than msi. On the downside, it doesn't work with virtualenvs. > > As regards eggs, they are essentially the precursor to wheels as a binary > distribution format. Users still using easy_install will need eggs (but why > aren't they using pip these days?). It's possible that older systems such > as buildout may still need eggs, but I'm not familiar with these systems > and don't know if they can use wheels instead these days. Buildout is based on easy_install and doesn't support wheels. It supports .egg or, I believe, .exe installers. I'm not sure about .msi. If setuptools were to support wheels, buildout would get that support automatically, AFAIU. (Why doesn't setuptools support wheels natively?) Wheel files can be created automatically from .egg/.exe files using 'wheel convert', so supporting both shouldn't be a big hassle and can be automated. Marius Gedminas -- 2B OR NOT 2B == FF -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From marius at pov.lt Fri Nov 28 09:40:53 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 28 Nov 2014 10:40:53 +0200 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: Message-ID: <20141128084053.GB3505@fridge.pov.lt> On Fri, Nov 28, 2014 at 04:03:59PM +1000, Nick Coghlan wrote: > Here's my proposed change: > > ================= > The default platform tag is distutils.util.get_platform() with all > hyphens - and periods . replaced with underscore _ . If > /etc/os-release [N] exists on the system, then the values in the 'ID' > and 'VERSION_ID' fields are read, all hyphens - and periods . replaced > with underscore _ , and the results appended to the default tag after > a separating underscore." > > Examples: > > * win32 > * macosx_10_6_intel > * linux_x86_64_fedora_20 > * linux_x86_64_rhel_7_0 > * linux_x86_64_debian_7_0 > * linux_x86_64_ubuntu_14_04 > ================= > > The [N] reference would then be a reference to > http://www.freedesktop.org/software/systemd/man/os-release.html for > the definition of the format of os-release. (Note that while the > format originated with systemd, plenty of distros have also started > providing it regardless of which init system they use) ... > This also won't help with older Linux distros that don't offer > /etc/os-release, but I'm OK with that - those can just continue to > show up as "linux_x86_64", and PyPI can continue to disallow those > uploads. I was curious about what "older distros" meant in the context of Ubuntu, so I looked it up: /etc/os-release exists in Ubuntu 12.04 LTS (and newer) but didn't exist in Ubuntu 10.04 LTS. Support for Ubuntu 10.04 LTS ends in April 2015. Marius Gedminas -- Give a man a computer program and you give him a headache, but teach him to program computers and you give him the power to create headaches for others for the rest of his life... -- R. B. Forest -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From p at 2014.dobrogost.net Fri Nov 28 09:47:58 2014 From: p at 2014.dobrogost.net (Piotr Dobrogost) Date: Fri, 28 Nov 2014 09:47:58 +0100 Subject: [Distutils] What binary formats should we be distributing? In-Reply-To: References: <54776399.708@level12.io> Message-ID: On Fri, Nov 28, 2014 at 12:25 AM, Paul Moore wrote: > > The wininst (exe) format is probably of limited value, It is of use for > systems without pip where integration with add/remove programs is useful, > but such systems are probably rare. But if you want to distribute a > "standalone installer" format, exe is probably more familiar to Python > package users than msi. On the downside, it doesn't work with virtualenvs. It does work with virtualenv although only when used with easy_install and not pip. See http://stackoverflow.com/q/3271590/95735 Regards, Piotr Dobrogost From ncoghlan at gmail.com Fri Nov 28 16:27:44 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Nov 2014 01:27:44 +1000 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: <20141128091951.0ab5b751@fsol> References: <20141128091951.0ab5b751@fsol> Message-ID: On 28 November 2014 at 18:19, Antoine Pitrou wrote: > On Fri, 28 Nov 2014 16:03:59 +1000 > Nick Coghlan wrote: >> Here's my proposed change: >> >> ================= >> The default platform tag is distutils.util.get_platform() with all >> hyphens - and periods . replaced with underscore _ . If >> /etc/os-release [N] exists on the system, then the values in the 'ID' >> and 'VERSION_ID' fields are read, all hyphens - and periods . replaced >> with underscore _ , and the results appended to the default tag after >> a separating underscore." >> >> Examples: >> >> * win32 >> * macosx_10_6_intel >> * linux_x86_64_fedora_20 >> * linux_x86_64_rhel_7_0 >> * linux_x86_64_debian_7_0 >> * linux_x86_64_ubuntu_14_04 > > Is this not going to be a slippery slope? Only if folks publish Linux binaries themselves, and that's still a bad idea (for the same reason publishing distro binaries is already a rare thing for people to do). >> Now, this slightly overspecifies on the *consumer* side. A binary >> wheel that works on "rhel_7_0" for example, should almost certainly >> work on "rhel_7_1". However, that can be addressed on the tooling side >> (e.g. permitting the specification of "additional compatible >> platforms" when invoking pip), rather than needing to be in the >> specification. > > How about those lesser known distributions (e.g. Linux Mint or Mageia)? They tend to publish /etc/os-release as well these days, and there's actually a mechanism built into that for clients to flag other distros to try. > How many binary packages will package authors have to provide to cover > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied > by three or four Python versions is already a lot of binaries to > build... I'd still advise against folks posting Linux wheels on PyPI, just as they tend not to post RPM or deb files. This is so we can provide wheels at the distro level (or build them internally) without creating vast amounts of confusion. > While this would be a good technical solution, I think it's socially > disastrous. Only if you're expecting folks to regularly publish their own wheels to PyPI. This isn't really about that - it's about having a way to tackle it at the distro level, without introducing significant potential for confusion on end user systems (https://fedoraproject.org/wiki/Env_and_Stacks/Projects/LanguageSpecificRepositories describes the Fedora side of our current work in this area) > Of course, you may point out that it has its roots in the failure of the > GNU/Linux ecosystem to provide real binary compatibility. It's stunning > that under Windows you can build a Windows XP-compatible shared library > with a recent MSVC just by turning a switch in the options... The difference isn't really that surprising - both Microsoft and Apple have relied heavily on intellectual monopoly laws to retain control of their ecosystems. You can do a lot to constrain the choices of others when you have the full weight of the US government and copyright industry behind you. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Nov 28 16:38:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Nov 2014 01:38:55 +1000 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: <5478955C.2030209@ubuntu.com> References: <5478955C.2030209@ubuntu.com> Message-ID: On 29 November 2014 at 01:31, Matthias Klose wrote: > On 11/28/2014 07:03 AM, Nick Coghlan wrote: >> >> We've discussed the idea of changing the wheel file naming scheme to >> deal with Linux previously, but never put together a concrete >> proposal. >> >> The closest we've got is the idea of allowing the platform tag to be >> customised in pip and perhaps bdist_wheel, and while that's good from >> an "enabling experimentation" perspective, it may be overkill if the >> primary goal is just to better support handling of Linux distros. >> >> For starters, here's the current definition of the platform tag in PEP >> 425: > > hmm, maybe you repeat the rationale here for starters? To be able to publish wheel files for a particular ecosystem, without causing confusion if those wheel files somehow end up in an unsuitable environment (e.g. a Fedora specific wheel ending up on a Debian machine). >> ================= >> The platform tag is simply distutils.util.get_platform() with all >> hyphens - and periods . replaced with underscore _ . >> >> * win32 >> * linux_i386 >> * linux_x86_64 >> ================= > > > this is already wrong for ARM32 soft-float and hard-float, and > x86_64-linux-gnu and x86_64-linux-gnux32. If something is changed, then > please change that as well, maybe using something already defined like the > multiarch triplet. I'm open to completely redefining this in a distutils independent way, but it will need someone to define the precise algorithm. For the cases I'm personally worried about (i.e. Fedora & EPEL), the existing information extraction from os.uname() should be adequate. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From doko at ubuntu.com Fri Nov 28 16:31:40 2014 From: doko at ubuntu.com (Matthias Klose) Date: Fri, 28 Nov 2014 16:31:40 +0100 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: Message-ID: <5478955C.2030209@ubuntu.com> On 11/28/2014 07:03 AM, Nick Coghlan wrote: > We've discussed the idea of changing the wheel file naming scheme to > deal with Linux previously, but never put together a concrete > proposal. > > The closest we've got is the idea of allowing the platform tag to be > customised in pip and perhaps bdist_wheel, and while that's good from > an "enabling experimentation" perspective, it may be overkill if the > primary goal is just to better support handling of Linux distros. > > For starters, here's the current definition of the platform tag in PEP 425: hmm, maybe you repeat the rationale here for starters? > ================= > The platform tag is simply distutils.util.get_platform() with all > hyphens - and periods . replaced with underscore _ . > > * win32 > * linux_i386 > * linux_x86_64 > ================= this is already wrong for ARM32 soft-float and hard-float, and x86_64-linux-gnu and x86_64-linux-gnux32. If something is changed, then please change that as well, maybe using something already defined like the multiarch triplet. From solipsis at pitrou.net Fri Nov 28 16:51:58 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 28 Nov 2014 16:51:58 +0100 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: <20141128091951.0ab5b751@fsol> Message-ID: <20141128165158.096f5f26@fsol> On Sat, 29 Nov 2014 01:27:44 +1000 Nick Coghlan wrote: > > > > Is this not going to be a slippery slope? > > Only if folks publish Linux binaries themselves, and that's still a > bad idea (for the same reason publishing distro binaries is already a > rare thing for people to do). Well, let's not make this a matter of ideology. Everyone knows it's a bad idea to publish binaries, yet it's often better than nothing, especially if the software is tedious to compile. > > How many binary packages will package authors have to provide to cover > > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied > > by three or four Python versions is already a lot of binaries to > > build... > > I'd still advise against folks posting Linux wheels on PyPI, just as > they tend not to post RPM or deb files. This is so we can provide > wheels at the distro level (or build them internally) without creating > vast amounts of confusion. So do we (software authors) have to wait for that mythical "we" who are going to build binaries in time for our packages? Case in point: can I ask you (the mythical "we") to build packages for all major distros (including supported LTS releases), and the four most recent Python versions, of the following piece of software: https://github.com/numba/llvmlite :-) > This isn't really about that - it's about having a way to > tackle it at the distro level, without introducing significant > potential for confusion on end user systems I'm not sure I understand: distros provide their own packages, they don't (shouldn't) blindly pull binary wheels from PyPI. Why would they depend on the wheel tagging format? > The difference isn't really that surprising - both Microsoft and Apple > have relied heavily on intellectual monopoly laws to retain control of > their ecosystems. You can do a lot to constrain the choices of others > when you have the full weight of the US government and copyright > industry behind you. That discussion is a bit off-topic, but I don't think it has anything to do with copyright (and from I've seen in python-dev discussions, I'm not sure Apple is a good example). Regards Antoine. From ncoghlan at gmail.com Sat Nov 29 16:47:16 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 01:47:16 +1000 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: <20141128165158.096f5f26@fsol> References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> Message-ID: On 29 November 2014 at 01:51, Antoine Pitrou wrote: > On Sat, 29 Nov 2014 01:27:44 +1000 > Nick Coghlan wrote: >> > >> > Is this not going to be a slippery slope? >> >> Only if folks publish Linux binaries themselves, and that's still a >> bad idea (for the same reason publishing distro binaries is already a >> rare thing for people to do). > > Well, let's not make this a matter of ideology. Everyone knows it's a > bad idea to publish binaries, yet it's often better than nothing, > especially if the software is tedious to compile. It's not a matter of ideology, but a matter of practicality. Debian stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow enough (and have strong enough ABI compatibility guarantees) to be practical for ISVs to target with prebuilt binaries. Beyond that, the rate of development and breadth of target environments in the Linux world tends to make providing prebuilt binaries in *any* format difficult (unless you statically link everything you need, which works fine until you decide you want a particular package to depend on binaries provided by another one by linking to them dynamically). Regardless of target environment, though, being able to prebuild wheel files in a way that clearly indicates the platform restricted nature of the end result is a useful tool for system integrators. That benefit applies regardless of whether the builds are happening in the context of Linux distro development, or in the context of maintaining a particular set of infrastructure services. >> > How many binary packages will package authors have to provide to cover >> > people's needs? Windows + OS X + Linux multiplied by 32 / 64 multiplied >> > by three or four Python versions is already a lot of binaries to >> > build... >> >> I'd still advise against folks posting Linux wheels on PyPI, just as >> they tend not to post RPM or deb files. This is so we can provide >> wheels at the distro level (or build them internally) without creating >> vast amounts of confusion. > > So do we (software authors) have to wait for that mythical "we" who are > going to build binaries in time for our packages? If the compatibility tagging issue can be resolved, I don't think there should be any restrictions against uploading Linux wheels that avoid the generic Linux platform tags. I just doubt it will make sense for most developers to worry about it, just as most of them don't worry about providing RPMs or Debian packages. > Case in point: can I ask you (the mythical "we") to build packages for > all major distros (including supported LTS releases), and the four most > recent Python versions, of the following piece of software: > https://github.com/numba/llvmlite No, that would be a service provided by the as yet hypothetical PyPI build farm. If/when that happens, it will need to have a way of tagging Linux wheels appropriately, though. Nearer term (and what prompted me to start this thread), the Fedora Environments & Stacks working group is investigating providing prebuilt wheel files for the Fedora ecosystem, and potentially for EPEL as well (see https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement for the broader context of that effort). For other ecosystems, you'll have to ask participants in those ecosystems. >> This isn't really about that - it's about having a way to >> tackle it at the distro level, without introducing significant >> potential for confusion on end user systems > > I'm not sure I understand: distros provide their own packages, they > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they > depend on the wheel tagging format? We don't plan to blindly pull anything from PyPI - we're looking at the feasibility of publishing already reviewed software in ecosystem native formats (with the two pilot projects focusing on Java JAR files and Python wheel files). When I last mentioned that idea here, Marcus pointed out that doing that with the generic "linux_x86_64" compatibility tag on the wheel filenames would be problematic, as there'd be nothing preventing anyone from pulling them down onto inappropriate systems, with no obvious trace of the Fedora or EPEL specific assumptions that went into building them. While that's a valid concern, I also don't want to go invent our own custom compatibility tagging convention just for Fedora & EPEL, but rather work within the limits of what upstream Python packaging natively supports. At the moment there is no such tagging system, which is why I'm interested in pursuing improvements to the definition of the platform tag. However, after thinking further about the situation with EPEL (where we'd likely want a single set of wheel files to cover not only an entire major release series, but also downstream RHEL derivatives like CentOS), the possibility of sharing wheel files between the Fedora & Software Collections Python builds, and the point Matthias raised about the limitations of the current platform tag when it comes to multiarch support on Debian, I'm back to considering the idea of making it possible to override the default platform tag with something more appropriate (I think Daniel may have been the first one to suggest that?). The task of defining what the appropriate platform tag overrides should be would then fall back on the Linux distros as part of declaring our ABI compatibility expectations. Slightly tangentially, I'm also now wondering if we might also be able to address the SOABI problem on Python 2 with an appropriate configuration system design. Consider if the following could be included in the "pyvenv.cfg" file in a virtual environment: [compatibility] python=cp27,cp2,py2 abi=cp27mu platform=linux_x86_64_epel_6 Or for a Python 3 virtual environment: [compatibility] python=cp34,cp3,py3 abi=cp34m,abi3 platform=linux_x86_64_epel_6 If present, these pyvenv.cfg settings would override the normal PEP 425 compatibility tag calculations (I'm OK with the idea of *needing* to be in a virtual environment to gain the power to configure these tags). Note that this would *only* affect the tags used in filenames, and when searching for wheel files. It would still be up I'm not sold on the technical details yet (consider the above to be just a sketch of the idea), but it seems to me that having those virtual environment specific overrides defined in a tool independent way, and updating bdist_wheel to read them when creating wheel files, and pip to read them when deciding which wheel files to consider as candidates for installation, would likely address the use case, without locking us in to anything too specific in the absence of suitable data. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Sat Nov 29 17:10:29 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Nov 2014 17:10:29 +0100 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> Message-ID: <20141129171029.40fb6899@fsol> On Sun, 30 Nov 2014 01:47:16 +1000 Nick Coghlan wrote: > On 29 November 2014 at 01:51, Antoine Pitrou wrote: > > On Sat, 29 Nov 2014 01:27:44 +1000 > > Nick Coghlan wrote: > >> > > >> > Is this not going to be a slippery slope? > >> > >> Only if folks publish Linux binaries themselves, and that's still a > >> bad idea (for the same reason publishing distro binaries is already a > >> rare thing for people to do). > > > > Well, let's not make this a matter of ideology. Everyone knows it's a > > bad idea to publish binaries, yet it's often better than nothing, > > especially if the software is tedious to compile. > > It's not a matter of ideology, but a matter of practicality. Debian > stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow > enough (and have strong enough ABI compatibility guarantees) to be > practical for ISVs to target with prebuilt binaries. It seems we disagree on the notion of "practicality" :-) For me practicality means being able to build a single binary package for all recent Linux distros in a best effort approach. Building a different package for each distro version is far from practical for any reasonable-sized project (i.e. not something sponsored by a 1000+-employee entity, with a dedicated build team and infrastructure). > > Case in point: can I ask you (the mythical "we") to build packages for > > all major distros (including supported LTS releases), and the four most > > recent Python versions, of the following piece of software: > > https://github.com/numba/llvmlite > > No, that would be a service provided by the as yet hypothetical PyPI > build farm. If/when that happens, it will need to have a way of > tagging Linux wheels appropriately, though. "If/when that happens" is not reassuring, especially in the light of how many pie-in-the-sky improvements in the packaging ecosystems have turned out :-/ (at Continuum we have started offering such a service, but it's "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix) > Nearer term (and what prompted me to start this thread), the Fedora > Environments & Stacks working group is investigating providing > prebuilt wheel files for the Fedora ecosystem, and potentially for > EPEL as well (see > https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement > for the broader context of that effort). For other ecosystems, you'll > have to ask participants in those ecosystems. That's asking software authors to complicate and slow down their development process a lot. Also, there's no guarantee that Fedora or Ubuntu or whatever would actually *accept* to help us, right? > > I'm not sure I understand: distros provide their own packages, they > > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they > > depend on the wheel tagging format? > > We don't plan to blindly pull anything from PyPI - we're looking at > the feasibility of publishing already reviewed software in ecosystem > native formats (with the two pilot projects focusing on Java JAR files > and Python wheel files). > > When I last mentioned that idea here, Marcus pointed out that doing > that with the generic "linux_x86_64" compatibility tag on the wheel > filenames would be problematic, as there'd be nothing preventing > anyone from pulling them down onto inappropriate systems, with no > obvious trace of the Fedora or EPEL specific assumptions that went > into building them. Uh, there's a lot of hidden knowledge required to understand those two paragraphs that I don't master. I don't know what "inappropriate systems" are, what are "reviewed software", etc. ;-) Also I don't understand why you're not recompiling as you would normally do. > While that's a valid concern, I also don't want to go invent our own > custom compatibility tagging convention just for Fedora & EPEL, but > rather work within the limits of what upstream Python packaging > natively supports. Well, *allowing* distro tags in the platform tag is certainly ok. What I'm afraid of is if that's made mandatory. > Consider if the following could be included in the "pyvenv.cfg" file > in a virtual environment: > > [compatibility] > python=cp27,cp2,py2 > abi=cp27mu > platform=linux_x86_64_epel_6 > > Or for a Python 3 virtual environment: > > [compatibility] > python=cp34,cp3,py3 > abi=cp34m,abi3 > platform=linux_x86_64_epel_6 > > If present, these pyvenv.cfg settings would override the normal PEP > 425 compatibility tag calculations (I'm OK with the idea of *needing* > to be in a virtual environment to gain the power to configure these > tags). As long as it's on an opt-in basis, it certainly sounds ok. Regards Antoine. From ncoghlan at gmail.com Sat Nov 29 18:02:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Nov 2014 03:02:57 +1000 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: <20141129171029.40fb6899@fsol> References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> <20141129171029.40fb6899@fsol> Message-ID: On 30 November 2014 at 02:10, Antoine Pitrou wrote: > On Sun, 30 Nov 2014 01:47:16 +1000 > Nick Coghlan wrote: >> On 29 November 2014 at 01:51, Antoine Pitrou wrote: >> > On Sat, 29 Nov 2014 01:27:44 +1000 >> > Nick Coghlan wrote: >> >> > >> >> > Is this not going to be a slippery slope? >> >> >> >> Only if folks publish Linux binaries themselves, and that's still a >> >> bad idea (for the same reason publishing distro binaries is already a >> >> rare thing for people to do). >> > >> > Well, let's not make this a matter of ideology. Everyone knows it's a >> > bad idea to publish binaries, yet it's often better than nothing, >> > especially if the software is tedious to compile. >> >> It's not a matter of ideology, but a matter of practicality. Debian >> stable, RHEL/CentOS, Ubuntu LTS, SLES - distros like these move slow >> enough (and have strong enough ABI compatibility guarantees) to be >> practical for ISVs to target with prebuilt binaries. > > It seems we disagree on the notion of "practicality" :-) Note I said "ISVs" there - folks that actually make money from targetting those platforms. I don't think providing integrated Linux binaries is currently practical at all for open source projects (note that even CPython just provides a tarball upstream, with the builds handled directly by the distros). > For me practicality means being able to build a single binary package > for all recent Linux distros in a best effort approach. That isn't realistically possible for anything that isn't completely statically linked, as distro level choices of build options vary too much. We could technically allow statically linked binaries on PyPI by dropping the restriction against generic Linux wheel files, but I'm wary of that in the absence of an automated server side scan for dynamically linked binaries. Many users (quite reasonably, if they're primarily Python developers) have problems working through build failures when attempting to install non-Python extensions from source. Such build failures are usually models of clarity compared to diagnosing dynamic linking failures. > Building a > different package for each distro version is far from practical for any > reasonable-sized project (i.e. not something sponsored by a > 1000+-employee entity, with a dedicated build team and infrastructure). This is why I'm saying people *shouldn't* try to provide prebuilt Linux binaries in the general case: it's simply not practical. Hence the rise of higher level, more self-contained formats like Docker, as well as the fact that I'm not even trying to solve the "any Linux" problem myself, but going after the vastly simpler problem of targeting the ecosystem I need to target (but hopefully in a sufficiently general purpose way that other ecosystems can adopt the same model if they so choose). >> > Case in point: can I ask you (the mythical "we") to build packages for >> > all major distros (including supported LTS releases), and the four most >> > recent Python versions, of the following piece of software: >> > https://github.com/numba/llvmlite >> >> No, that would be a service provided by the as yet hypothetical PyPI >> build farm. If/when that happens, it will need to have a way of >> tagging Linux wheels appropriately, though. > > "If/when that happens" is not reassuring, especially in the light of > how many pie-in-the-sky improvements in the packaging ecosystems have > turned out :-/ Nobody is currently offering to provide paid staff to work on it, so it won't happen until it gets to the top of volunteers' personal priority lists. > (at Continuum we have started offering such a service, but it's > "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix) Yes, Continuum avoided the distro ABI compatibility problem by defining its own ABI. It's exactly the same model I'm proposing - in the general case, you can't take Continuum built packages, and use them with Fedora/Debian/etc built dynamically linked dependencies. Same with my suggestion - it's about labelling the compatibility requirements, rather than trying to avoid them by statically linking everything. >> Nearer term (and what prompted me to start this thread), the Fedora >> Environments & Stacks working group is investigating providing >> prebuilt wheel files for the Fedora ecosystem, and potentially for >> EPEL as well (see >> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/UserLevelPackageManagement >> for the broader context of that effort). For other ecosystems, you'll >> have to ask participants in those ecosystems. > > That's asking software authors to complicate and slow down their > development process a lot. If the distro level service doesn't benefit them, there's no reason for software authors to use it. This is about adding a new option for consuming software, not taking anything away. In my case, I won't deploy something until it's been through at least a basic licensing review, and if we're doing that work anyway, I may as well see if we can find a way to do it upstream in Fedora rather than inside the Red Hat firewall. Same for prebuilding binary wheels - sure, we could do that just for our own internal infrastructure, but it makes more sense to me to at least investigate the idea of handling it upstream first. > Also, there's no guarantee that Fedora or > Ubuntu or whatever would actually *accept* to help us, right? For Fedora, myself and Slavek (the Fedora Python maintainer) are the two folks working on this (& we're both voting members of the Environments & Stacks working group). So yes, in Fedora's case, it's definitely a developer experience problem we want to solve properly, and we're already working on it - this is me asking upstream for help resolving a design problem, rather than coming up with a speculative idea to persuade Fedora to adopt later. I realise that's very different from the historical attitude of Linux distributions to upstream packaging ecosystems. Getting to this point of adopting a more user-focused mindset when asking what services a Linux distro should be offering has been a formidable political challenge involving a lot of work from a lot of different people (including the current Fedora Project Leader), so I'm not even going to speculate on what might be involved in attempting to replicate that in the context of a different distro ecosystem :) >> > I'm not sure I understand: distros provide their own packages, they >> > don't (shouldn't) blindly pull binary wheels from PyPI. Why would they >> > depend on the wheel tagging format? >> >> We don't plan to blindly pull anything from PyPI - we're looking at >> the feasibility of publishing already reviewed software in ecosystem >> native formats (with the two pilot projects focusing on Java JAR files >> and Python wheel files). >> >> When I last mentioned that idea here, Marcus pointed out that doing >> that with the generic "linux_x86_64" compatibility tag on the wheel >> filenames would be problematic, as there'd be nothing preventing >> anyone from pulling them down onto inappropriate systems, with no >> obvious trace of the Fedora or EPEL specific assumptions that went >> into building them. > > Uh, there's a lot of hidden knowledge required to understand those > two paragraphs that I don't master. I don't know what "inappropriate > systems" are, what are "reviewed software", etc. ;-) An example of inappropriate systems: trying to install a Fedora wheel on Ubuntu. That often won't work in the general case, but pip will currently allow it (because they share a platform tag, despite not sharing a platform ABI) "Reviewed software" just refers to the Fedora package set. Currently we only make that available for installation as RPMs, the Environments & Stacks WG is looking at making it available in more developer friendly formats, rather than focusing solely on the operations use case. (We're not sure yet how we'll make that sustainable from a build infrastructure perspective, but that's a large part of what the pilot projects are aiming to figure out) > Also I don't understand why you're not recompiling as you would > normally do. We'll still be recompiling. We just want to publish the results for Python packages as virtual-environment compatible wheel files, not just as RPMs. >> While that's a valid concern, I also don't want to go invent our own >> custom compatibility tagging convention just for Fedora & EPEL, but >> rather work within the limits of what upstream Python packaging >> natively supports. > > Well, *allowing* distro tags in the platform tag is certainly ok. What > I'm afraid of is if that's made mandatory. OK, that makes more sense. Yes, I agree we need to keep the ability to say "this is a prebuilt, self-contained, binary wheel that should run on any Linux system because it doesn't link to any system binaries". Chalk it up as yet another reason that the specific proposal I started the thread with wouldn't actually work :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tseaver at palladion.com Sat Nov 29 18:39:18 2014 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 29 Nov 2014 12:39:18 -0500 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: <20141129171029.40fb6899@fsol> References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> <20141129171029.40fb6899@fsol> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/29/2014 11:10 AM, Antoine Pitrou wrote: > For me practicality means being able to build a single binary package > for all recent Linux distros in a best effort approach. I can't imagine finding any such binary useful (assuming a none-pure-Python project): the chance that it *might* blow up makes it easier just to create my own from the sdist. Such binaries would tend to bitrot, even if they were OK for the "supported" set of distros at the time they were made. How can such wheels be feasible, when cross-distro RPMs / .debs are clearly not (in the general sense)? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlR6BMUACgkQ+gerLs4ltQ70sgCeJ8BmRVDOhEzJ4I70eGWQCLn4 l0cAoJwoHcC26M/t5mhLXgJl4IrugDcV =G5zd -----END PGP SIGNATURE----- From solipsis at pitrou.net Sat Nov 29 18:41:37 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Nov 2014 18:41:37 +0100 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> <20141129171029.40fb6899@fsol> Message-ID: <20141129184137.316026d6@fsol> On Sun, 30 Nov 2014 03:02:57 +1000 Nick Coghlan wrote: > Many users (quite reasonably, if they're primarily Python developers) > have problems working through build failures when attempting to > install non-Python extensions from source. Such build failures are > usually models of clarity compared to diagnosing dynamic linking > failures. However, installing a binary doesn't imply a potential longish building step, or the installation of many build dependencies. LLVM can take 20 minutes to compile on a modern quad-core x86. I've been told it takes several hours on a Cortex A8 platform... By comparison, the failure of loading a precompiled dynamic library is instantaneous. And I don't think build failures are understandable by many users. You need to be a seasoned C developer for that. > > (at Continuum we have started offering such a service, but it's > > "generic Linux": http://docs.binstar.org/build-config.html#BuildMatrix) > > Yes, Continuum avoided the distro ABI compatibility problem by > defining its own ABI. Not exactly. Some ABI problems - for example the glibc-related ones - are still here. Conda and binstar-build are still a best effort (on the GNU/Linux side, that is), not an ideal solution. > > Well, *allowing* distro tags in the platform tag is certainly ok. What > > I'm afraid of is if that's made mandatory. > > OK, that makes more sense. Yes, I agree we need to keep the ability to > say "this is a prebuilt, self-contained, binary wheel that should run > on any Linux system because it doesn't link to any system binaries". > Chalk it up as yet another reason that the specific proposal I started > the thread with wouldn't actually work :) Great! Regards Antoine. From donald at stufft.io Sat Nov 29 18:40:00 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 12:40:00 -0500 Subject: [Distutils] Proposal: using /etc/os-release in the "platform tag" definition for wheel files In-Reply-To: References: <20141128091951.0ab5b751@fsol> <20141128165158.096f5f26@fsol> <20141129171029.40fb6899@fsol> Message-ID: <8D409CE6-F5B1-430A-8B84-10592C96111F@stufft.io> Binary compatibility is not something that is my strong suite, however I think that whatever we do should be aimed at eventual support of uploading Linux binaries to PyPI. Currently we allow Windows and OSX and from what I understand the same Linux ABI problems can happen in those situations, it's just less likely because of the single vendor nature of those platforms. In that vein I don't believe we need to *solve* the Linux ABI problem generically but that we need to get something that will most likely work. To this end the idea of using the distro name and version sounds like a solution that will get the Linux side of things closer to the Windows and OS X side. I do think we should continue to support the plain linux values and possibly we should even allow them to be uploaded to PyPI. I had also previously though about an additional file in the metadata spec which would encode the SOABI of each thing that has been dynamically linked (it's my understanding that you can inspect this?) and then when pip downloads a Wheel file it can use that file as a final and certain check of compatability. In this way the filename would only be used to try and select the file which is most likely to work on the system but that we can detect once we've downloaded the file if it will or will not work and fall back to attempt other files, possibly even the sdist. I don't think we need to try and protect people from uploading a badly built Wheel. If a project uploads a Wheel that has incorrect platform tags then that is a bug in their project as long as we provide reasonable differentiators that people can use to tag their Wheels with. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sun Nov 30 01:43:25 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 29 Nov 2014 19:43:25 -0500 Subject: [Distutils] API CHANGE - Migrating from MD5 to SHA2, Take 2 In-Reply-To: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> References: <54F34BF5-1A4B-4452-AC2D-6EE9D837074B@stufft.io> Message-ID: <2EC8ACAD-7E01-4B57-BD79-4882368CE04F@stufft.io> > On Nov 13, 2014, at 9:21 PM, Donald Stufft wrote: > > Starting a new thread with more explicit details at Richard?s request. > Essentially the tl;dr here is that we'll switch to using sha2 (specifically > sha256). Ping? Are we OK to make this change? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA