From matthew.brett at gmail.com Thu Sep 18 17:32:28 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Thu, 18 Sep 2014 11:32:28 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, I'd very much like your feedback on a utility I've written to make OSX installers from wheels. For my own course, and for matplotlib, I wanted to make an double click OSX installer which would install a series of requirements into Python.org Python. As usual, I got a lot of useful thinking on this from Min Ragan-Kelly of IPython fame. I've written an package that automates building the installer: https://github.com/matthew-brett/wheels2dmg There are some examples in the README, but the basic idea is: wheels2dmg scipy-stack 1.0 scipy matplotlib ipython[notebook] \ pandas numexpr sympy --python-version 2.7.8 This uses pip to download wheels for the stated requirements, packages them up into a 'wheels' directory, creates an OSX ``.pkg`` installer that will install the wheels using pip, and puts this all into a ``.dmg`` disk image with a suitable README, and a link for downloading Python.org Python. The generated README covers installing into Macports and Homebrew. My hope is that this makes it easy for people who want to provide a whole stack to provide a ``.dmg`` installer that does the right thing in terms of updating dependencies. I'd love to get any feedback on whether this works for people, and missing features. Cheers, Matthew From nad at acm.org Thu Sep 18 21:21:15 2014 From: nad at acm.org (Ned Deily) Date: Thu, 18 Sep 2014 12:21:15 -0700 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements References: Message-ID: In article , Matthew Brett wrote: > I'd very much like your feedback on a utility I've written to make OSX > installers from wheels. > > For my own course, and for matplotlib, I wanted to make an double click OSX > installer which would install a series of requirements into Python.org Python. Thanks for looking at this. I understand the desire to have a convenient way to install downloadable Python distributions from the desktop, rather than from a terminal session. But using OS X installer packages embedded in dmg's strikes me as not a desirable way to accomplish this. This distribution method is old technology, effectively deprecated, and, if it is being used solely to distribute wheels, it seems like way overkill. The primary use case for installer packages is when you need to install files in various disparate parts of the file system, particularly in system areas that require admin privs to do so. Wheels are themselves a packaging format, managed nicely by pip. Bundling that into a another packaging format, one that doesn't even provide an uninstall capability unlike pip, seems wasteful and even confusing. And then embedding *that* in a dmg just makes it more complicated, what with mounting images and dismounting and deleting the dmg. If the installer package is a 10.5+ flat package, there's no need to embed it in an dmg; that was only necessary for the original, bundle format installer packages, which are a collection of directory trees with files, rather than a compressed container file. (Curiously enough, I'm working right now on finally transitioning the python.org installers away from bundle packages with dmgs to flat packages.) Another issue with installer packages is that they need to be signed by someone with a Mac Developer account from Apple to avoid users having to figure out to override Gatekeeper security settings (control-click -> Open (Installer), rather than double click), a less than ideal user experience. Ideally, IMO, I would like to see us come up with a Python.app that acts as a double-clickable GUI front end to Python for OS X. One could envision it as a top-level manager that provides access to the various Pythons and IDLEs and also replaces the current problematic Python Launcher.app. It could handle wheels by registering the .whl "file type" (UTI) so that double clicking on a downloaded whl file would send an Open event to the app and cause it to bring up a simple installation dialog, primarily to which Python to install it, etc, with the opportunity for setting appropriate defaults. The Python.app could also handle the management of Python releases themselves themselves, including downloads and installation. (Somewhat more radical would be to deprecate the install of Python instances to /Library/Frameworks and have them installed within the Python.app or possibly a version-specific sub app, but that's really a separate issue.) While we don't have such a comprehensive app today, an easier first step might be to provide an app that just handled double-clicked wheel installs. Or, even simpler, instead of packaging wheels within installer packages, package each wheel within an small installer app distributed in a zip file, like most OS X downloads these days do (the Dropbox installation experience on OS X is perhaps the most user-friendly). In either case, the apps would need to be signed. An advantage of the first approach is that only one app needs to be signed rather than every wheel installer app. -- Ned Deily, nad at acm.org From nad at acm.org Fri Sep 19 00:08:48 2014 From: nad at acm.org (Ned Deily) Date: Thu, 18 Sep 2014 15:08:48 -0700 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: On Sep 18, 2014, at 14:03 , Aaron Meurer wrote: > > On Thu, Sep 18, 2014 at 2:21 PM, Ned Deily wrote: >> In article >> , >> Matthew Brett wrote: >>> I'd very much like your feedback on a utility I've written to make OSX >>> installers from wheels. >>> >>> For my own course, and for matplotlib, I wanted to make an double click OSX >>> installer which would install a series of requirements into Python.org Python. >> >> Thanks for looking at this. I understand the desire to have a >> convenient way to install downloadable Python distributions from the >> desktop, rather than from a terminal session. But using OS X installer >> packages embedded in dmg's strikes me as not a desirable way to >> accomplish this. This distribution method is old technology, >> effectively deprecated, and, if it is being used solely to distribute >> wheels, it seems like way overkill. The primary use case for installer > > Why do you say that it's deprecated? Can you reference some Apple docs > that say this? Not by Apple necessarily, but, IME, in the greater OS X ecosphere. As I mentioned below, I consider Dropbox to be leading the way here but I don't see that many third-party apps or packages any more that are installed using the naked Installer.app interface. Some do install flat packages under the covers, of course. > If you are distributing an "app" (a .app package that gets installed > in /Applications) it is overkill, because you can just distribute a > dmg with the app itself and have users drag it to /Applications. But > for files that need to be installed into an esoteric location on the > system, it is still supported. > >> packages is when you need to install files in various disparate parts of >> the file system, particularly in system areas that require admin privs >> to do so. Wheels are themselves a packaging format, managed nicely by > > But if a user downloads a wheel and double clicks it doesn't install. Today, yes, but the point of the installer apps or Python.app that I proposed is to be the target applications for wheel files so that double-clicking on a wheel file would cause that app to be launched and install it with necessary interaction. Without further investigation, I *think* that is technically possible but I haven't tried it yet. If that can't be made to work, then this would not be as attractive an option. And a wheel file isn't supposed to be installed in disparate parts of the file system: it's supposed to be following the Distutils installation schemes. In general, to install into the python.org system site-packages dir would require any installer app to authenticate for admin privs. But a wheel installer app could also provide a user-friendly option to install into the appropriate per-user location (~/Library/Python or ~/.local) without admin privs. While optional --user installs could probably be supported in .pkg installs, the smarts to do so would have to be duplicated in every package. > I guess an advantage of DMG is that it verifies itself, although in > this day and age that's not really as big of an issue. Signed package files should verify themselves as well, no? And there is the option for signed whl files although, as I understand it, that does not include a trust model. > At any rate, +1 for moving to flat (new-style) installers, since these > are the ones that you can actually build with the commands that ship > with the newest versions of OS X, and are the ones actually documented > in Apple's docs (although most of the stuff you'll find around the > internet refer to the old format). > > Most importantly, though, you can actually sign new-style installers > with a developer key (PLEASE do this. If the PSF can't afford a > $100/yr. Apple developer account then they've got bigger issues to > worry about). Well, yes, that's the main reason for moving to flat packages :) The two issues have been that building bundle installers is deeply ingrained in the existing python.org installer build process and that we have been supporting older OS X versions (e.g. 10.3) that do not support flat packages. We retired 10.3/10.4 installer support with Python 3.3 and now on track to retire that support with the next Python 2.7 release (2.7.9): 10.3/10.4 installer support is deprecated as of 2.7.7. > The obvious use-case here is users who don't want to touch the command > line. If you care about being user-friendly to the most technically > challenged users, you should 1, use a GUI, and 2, sign that GUI with a > developer key. It's not possible to have a GUI installer that > Gatekeeper doesn't block by default, because that's the way Gatekeeper > works. Yes, that's why I said the wheel installer or Python apps would need to be signed. -- Ned Deily nad at acm.org -- [] From matthew.brett at gmail.com Fri Sep 19 18:03:52 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Fri, 19 Sep 2014 12:03:52 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On 9/18/14, Ned Deily wrote: > In article > , > Matthew Brett wrote: >> I'd very much like your feedback on a utility I've written to make OSX >> installers from wheels. >> >> For my own course, and for matplotlib, I wanted to make an double click >> OSX >> installer which would install a series of requirements into Python.org >> Python. > > Thanks for looking at this. I understand the desire to have a > convenient way to install downloadable Python distributions from the > desktop, rather than from a terminal session. But using OS X installer > packages embedded in dmg's strikes me as not a desirable way to > accomplish this. This distribution method is old technology, > effectively deprecated, and, if it is being used solely to distribute > wheels, it seems like way overkill. The primary use case is for getting started with a basic package setup. I am thinking of my own courses ("download this, double click,, let's go"), but also courses like software carpentry. Here the students may never have used Terminal.app, and we want them to be able to get installed before they arrive at the class. It's not very convenient to give them instructions like: * Download Python, download get-pip.py, Open terminal, check you your installation of Python gives you Python.org Python, run ``python{3} get-pip.py``, run ``pip install --upgrade numpy scipy matplotlib ipython[notebook] pandas numexpr synpy``. * Download all the necessary dmg. and wheel files and double click to install them. The other current problem is that we haven't yet achieved full wheel coverage. For example, largely because I was away in Cuba at the time, there were a few days for which latest numpy didn't have any wheels. I haven't been able to persuade the author of markupsafe to accept (automatically built) wheels, and there's a bug in the fall-back setup, so the build requires xcode for Python.org Python, which I want to avoid. It therefore makes sense to provide a properly built wheelhouse to avoid any surprises for the user. Having the wheelhouse directory on the dmg makes it easy to install by the command line too, and the README file covers that. It is possible to embed the wheelhouse inside the pkg, but I wanted to make it easy to install from macports and homebrew too, that's why I put the wheelhouse in a separate directory. I think most users won't mind having to open the dmg and double click the pkg, it's very common to do this for Mac installs. Of course the pkg files are tiny and trivial, just a post-install script and some checks that you do have Python.org Python installed. They are really easy to build with a few wrapped command lines. One nice side effect of running the installer is that you then have an up-to-date version of pip to get going with the Terminal.app lifestyle. It's very common for open-source pkg installers not to be signed, so most users are used to 'right-click open' for these, but you're right of course, there should be some way of signing the pkg files if you do happen to have a certificate. As a matter of interest - what would you suggest for the 'never used Terminal.app, need a lot of packages installed for course' use-case? > Ideally, IMO, I would like to see us come up with a Python.app that acts > as a double-clickable GUI front end to Python for OS X. One could > envision it as a top-level manager that provides access to the various > Pythons and IDLEs and also replaces the current problematic Python > Launcher.app. It could handle wheels by registering the .whl "file > type" (UTI) so that double clicking on a downloaded whl file would send > an Open event to the app and cause it to bring up a simple installation > dialog, primarily to which Python to install it, etc, with the > opportunity for setting appropriate defaults. The Python.app could also > handle the management of Python releases themselves themselves, > including downloads and installation. (Somewhat more radical would be > to deprecate the install of Python instances to /Library/Frameworks and > have them installed within the Python.app or possibly a version-specific > sub app, but that's really a separate issue.) > > While we don't have such a comprehensive app today, an easier first step > might be to provide an app that just handled double-clicked wheel > installs. Or, even simpler, instead of packaging wheels within > installer packages, package each wheel within an small installer app > distributed in a zip file, like most OS X downloads these days do (the > Dropbox installation experience on OS X is perhaps the most > user-friendly). In either case, the apps would need to be signed. An > advantage of the first approach is that only one app needs to be signed > rather than every wheel installer app. Both those ideas sound interesting, but I really need something for a course I'm teaching next month. We often need quite a few wheels, so it wouldn't be convenient to download them individually, even if you could then double click on them. Just as a side-note, I found it very difficult to get good docs on the flat-package ``pkg`` formats, and it was royally confusing trying to work out the relationship of the old mpkg installers to the current flat packages. This was the summary of my researches: https://matthew-brett.github.io/pydagogue/legacy_package_redux.html Thanks for the feedback and thoughts, Matthew From matthew.brett at gmail.com Mon Sep 22 15:37:26 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 22 Sep 2014 09:37:26 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On 9/19/14, Christopher Barker wrote: > On Fri, Sep 19, 2014 at 9:03 AM, Matthew Brett > wrote: > >> The primary use case is for getting started with a basic package >> setup. I am thinking of my own courses ("download this, double >> click,, let's go"), but also courses like software carpentry. Here >> the students may never have used Terminal.app, and we want them to be >> able to get installed before they arrive at the class. >> > > well yes and no -- for software carpentry in particular, aren't tey all > about learning some command line tools, etc? in fact, even more than python > itself? My impression is that software carpentry have the same requirements as I do - they want the install done by the time the students show up. They are indeed going to show them how to do things on the command line, but they don't assume they already know how to use it before they arrive, hence the desire for a point-click installer. > And for my intro to python classes, I start with "you're going to need to > do at least a little bit at the command line -- let's get you all going > there. And frankly, that's easier on OS-X than Windows! > > That being said -- yes, nifty installers a nice. > > I really like Ned's ideas, but: > > What can we write it in? If I were doing it, I'd want use wxPython, becase > that's how I know to write GUIS -- but that's a pretty big dependency for > the first thing you need to have. > > Could we use PyObjC ? is that delivered with Apple's python installs? > > Or would it have to be a simple Cocoa app, so that it was small and compact > and had no non-standad dependencies? > > And whols goign t odo this? clealry Mathew wants somethign he can use now > (or soon), so if no one has teh time and inclination to get a beta version > of an installer our soon, then he might as well kee going with can work > now. I believe the project works as it is now. I will probably start with an automated build / test of a scipy stack installer on travis, when I get to better internet, maybe in a couple of weeks. > The truth is, we talked a an app to handle point and click installation of > eggs years ago, and no one ever wrote one. > >> * Download Python, download get-pip.py, > > There is talk of the pip bootstrapping code getting into the standard py2 > installers -- is that going to happen with the next release -- that would > eliminate one annoying step! > >> Open terminal, check you your > >> installation of Python gives you Python.org Python, > > > That should more or less be done if you installed the python.org build.... Right - the installer just checks that you have in fact already installed Python.org; if you haven't the installer gives you a relevant message. >> run ``python{3} > >> get-pip.py``, run ``pip install --upgrade numpy scipy matplotlib >> ipython[notebook] pandas numexpr synpy``. >> > > It might be nice to have a "scipy-stack" meta-package that would simply > depend on all these (or most of them) That would be nice on multiple > platforms. > >> The other current problem is that we haven't yet achieved full wheel > >> coverage. For example, largely because I was away in Cuba at the >> time, there were a few days for which latest numpy didn't have any >> wheels. > > > yeah, this is too bad -- it really seems that "the way" 'to distribute > binaries for python packages is as wheels on PyPI -- building an entire > other infrastructure really sucks. I hope that when we've ironed out the builds for the wheels, we'll develop a habit of releasing the wheels at exactly the same time as we make the full release, probably by hiding the release while uploading the wheels. The numpy delay was because of a bug in my automated builds that Ralf Gommers bravely fixed while I was away. For a few other wheels, it seems that some webby package maintainers have some anxiety about wheel installation, and I couldn't reassure them with tests on every platform I could think of. I suppose that will resolve, given time, and even more comprehensive automated testing. Maybe it would also be useful to have some definitive opinion from the pypa on the likely safety of OSX wheels that I could point to. >> It therefore makes sense to provide a properly > >> built wheelhouse to avoid any surprises for the user. >> > > sure. > > >> Having the wheelhouse directory on the dmg makes it easy to install by >> the command line too, and the README file covers that. > > > Is it that hard to point our nifty still-imaginary installation tool at a > wheelhouse? I actually have scipy-stack pip requirement files up on the nipy server, so it would be trivial to do this - something like: pip install --no-index --upgrade -r https://nipy.bic.berkeley.edu/scipy_installers/scipy-stack-1.0-plus.txt I was looking around Berkeley campus to find some more permanent hosting for this - it would be fairly easy to make a general wheelhouse for all our scientific course installers, where the course requirements would be a pip requirements file. But - I think we need just a little bit more infrastructure to make this work smoothly... Any ideas? > >> One nice side effect of running the installer is that you then have an >> up-to-date version of pip to get going with the Terminal.app >> lifestyle. >> > > indeed -- maybe if nothing else, a point and click installer for pip.... > > It's very common for open-source pkg installers not to be signed, so >> most users are used to 'right-click open' for these, > > > I still find most users choke on this step :-( Sure - it would be great to have a few funded developer accounts to fix that. Ned - any chance of something like that from the PSF? >> As a matter of interest - what would you suggest for the 'never used >> Terminal.app, need a lot of packages installed for course' use-case? > > I give them clear instructions -- pretty much what you had above --Though > I could improve them -- here's my latest: > > http://codefellows.github.io/sea-f2-python-sept14/supplements/python_for_mac.html Nice - thanks. It would be really good if we could come up with a standard page for this - something like a mcscipy site ? > [I haven't pushed the source to gitHub for a while -- but I"ll do that if > anyone want to contribute!] > > That is for a class that is general python, no numpy/scipy., etc. And > looking now, I probably should have bit more about opening the terminal in > a dir, etc. I tend to run through some of that in the first class live, and > had no problems with the class I'm doing now. The Windows folks -- that's > another story! > > It does mean that that first thing they need to do is figure out at least a > tiny bit about the command line -- but really -- they are going to need > that anyway. Yes - if you don't have much they need to install, then you can easily do it with pip in the class - otherwise the time and bandwidth can be limiting. > I did find that the latest OS-X has an (optional, and off by default, I > think) right click on a dir and get a menu item to open a terminal there -- > that's a good start. > > And I point them to: > > http://cli.learncodethehardway.org/book/ > > >> Both those ideas sound interesting, but I really need something for a >> course I'm teaching next month. >> > > yeah, there is that! > > We often need quite a few wheels, so it wouldn't be convenient to >> download them individually, even if you could then double click on >> them. >> > > but pointing them to a wheelhouse isn't too bad, though I can see the > appeal of something they can JustGetInstalled -- all of it, before learning > ANYTHING new. Thanks a lot for the useful reflections. If y'all do use the wheels2dmg installer and need any new features, find it doesn't work, please do let me know... Onward and upward, Matthew From charlie.clark at clark-consulting.eu Mon Sep 22 15:59:51 2014 From: charlie.clark at clark-consulting.eu (Charlie Clark) Date: Mon, 22 Sep 2014 15:59:51 +0200 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Am .09.2014, 15:37 Uhr, schrieb Matthew Brett : > Yes - if you don't have much they need to install, then you can easily > do it with pip in the class - otherwise the time and bandwidth can be > limiting. A local devpi server can work wonders there. Purely regarding wheels - as far as I can tell they're not perfect yet. We dropped them from openpyxl because we couldn't stop them packaging "kitchen sink". This has already disappointed one person but particularly for people who just want something installed, we only want to ship the package. Python's packaging remains a work in progress. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a D?sseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 From matthew.brett at gmail.com Mon Sep 22 16:10:04 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 22 Sep 2014 10:10:04 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On 9/22/14, Charlie Clark wrote: > Am .09.2014, 15:37 Uhr, schrieb Matthew Brett : > >> Yes - if you don't have much they need to install, then you can easily >> do it with pip in the class - otherwise the time and bandwidth can be >> limiting. > > A local devpi server can work wonders there. Sure, but that is a lot of setup for one morning of installs. The benefit of the point-click installer is that the student can download everything they need in their own time. > Purely regarding wheels - as far as I can tell they're not perfect yet. We > dropped them from openpyxl because we couldn't stop them packaging > "kitchen sink". This has already disappointed one person but particularly > for people who just want something installed, we only want to ship the > package. Would you mind saying more about what went wrong there? Wheels will install all the listed dependencies, but if there are dependencies you don't want, and are not necessary, then the package maintainer needs to know... That would be the same for wheels as for a source install, as far as I know. Cheers, Matthew From charlie.clark at clark-consulting.eu Mon Sep 22 16:15:59 2014 From: charlie.clark at clark-consulting.eu (Charlie Clark) Date: Mon, 22 Sep 2014 16:15:59 +0200 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Am .09.2014, 16:10 Uhr, schrieb Matthew Brett : > Sure, but that is a lot of setup for one morning of installs. The > benefit of the point-click installer is that the student can download > everything they need in their own time. devpi is pretty easy to setup but I understand your point-click, er, point. >> Purely regarding wheels - as far as I can tell they're not perfect yet. >> We >> dropped them from openpyxl because we couldn't stop them packaging >> "kitchen sink". This has already disappointed one person but >> particularly >> for people who just want something installed, we only want to ship the >> package. > Would you mind saying more about what went wrong there? Wheels will > install all the listed dependencies, but if there are dependencies you > don't want, and are not necessary, then the package maintainer needs > to know... That would be the same for wheels as for a source install, > as far as I know. It's not dependencies but not respecting the exclusions we use for sdist. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a D?sseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 From charlie.clark at clark-consulting.eu Mon Sep 22 16:28:36 2014 From: charlie.clark at clark-consulting.eu (Charlie Clark) Date: Mon, 22 Sep 2014 16:28:36 +0200 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Am .09.2014, 16:24 Uhr, schrieb Matthew Brett : > Sorry to be dumb, but I don't understand. Do you mean that you built > a source tarball of your own package and you tried to modify the > dependencies of other packages from that source tarball setup.py? Nothing to do with dependencies. > Was there any difference between source and wheel installation? Yes, the wheel includes tests and other code we don't have in the source installation. I don't know why this is the case as I, fortunately, don't manage releases and don't need to learn the pitfalls of any new formats. For us it was just easier to drop it. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a D?sseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 From matthew.brett at gmail.com Mon Sep 22 16:24:58 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 22 Sep 2014 10:24:58 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On 9/22/14, Charlie Clark wrote: > Am .09.2014, 16:10 Uhr, schrieb Matthew Brett : > >> Sure, but that is a lot of setup for one morning of installs. The >> benefit of the point-click installer is that the student can download >> everything they need in their own time. > > devpi is pretty easy to setup but I understand your point-click, er, point. > >>> Purely regarding wheels - as far as I can tell they're not perfect yet. >>> We >>> dropped them from openpyxl because we couldn't stop them packaging >>> "kitchen sink". This has already disappointed one person but >>> particularly >>> for people who just want something installed, we only want to ship the >>> package. > >> Would you mind saying more about what went wrong there? Wheels will >> install all the listed dependencies, but if there are dependencies you >> don't want, and are not necessary, then the package maintainer needs >> to know... That would be the same for wheels as for a source install, >> as far as I know. > > It's not dependencies but not respecting the exclusions we use for sdist. Sorry to be dumb, but I don't understand. Do you mean that you built a source tarball of your own package and you tried to modify the dependencies of other packages from that source tarball setup.py? Was there any difference between source and wheel installation? Cheers, Matthew From matthew.brett at gmail.com Mon Sep 22 16:42:05 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 22 Sep 2014 10:42:05 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On 9/22/14, Charlie Clark wrote: > Am .09.2014, 16:24 Uhr, schrieb Matthew Brett : > >> Sorry to be dumb, but I don't understand. Do you mean that you built >> a source tarball of your own package and you tried to modify the >> dependencies of other packages from that source tarball setup.py? > > Nothing to do with dependencies. > >> Was there any difference between source and wheel installation? > > Yes, the wheel includes tests and other code we don't have in the source > installation. I don't know why this is the case as I, fortunately, don't > manage releases and don't need to learn the pitfalls of any new formats. > For us it was just easier to drop it. So just to be clear, the difference here is between the instlal you got for: pip install your-package where `your-package` was either an sdist built with ``python setup.py sdist`` or a wheel built with ``python setup.py bdist_wheel``? Is your package online somewhere to have a look at the setup.py? My best guess would be you had some custom code for building the sdist that is not being run for other distutils build commands. If that's the case, then you could certainly reasonably blame the fact that distutils is complicated, but it would not be a general problem with wheels. Cheers, Matthew From charlie.clark at clark-consulting.eu Mon Sep 22 16:46:38 2014 From: charlie.clark at clark-consulting.eu (Charlie Clark) Date: Mon, 22 Sep 2014 16:46:38 +0200 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Am .09.2014, 16:42 Uhr, schrieb Matthew Brett : > So just to be clear, the difference here is between the instlal you got > for: > pip install your-package > where `your-package` was either an sdist built with ``python setup.py > sdist`` or a wheel built with ``python setup.py bdist_wheel``? > Is your package online somewhere to have a look at the setup.py? https://bitbucket.org/openpyxl/openpyxl/src > My best guess would be you had some custom code for building the sdist > that is not being run for other distutils build commands. If that's > the case, then you could certainly reasonably blame the fact that > distutils is complicated, but it would not be a general problem with > wheels. No, as I said the manifest wasn't being respected with wheels so we stopped using them. No idea whether this applies to any other projects or why it is. As I said before, Python's packaging is still a work in progress. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a D?sseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 From charlie.clark at clark-consulting.eu Tue Sep 23 15:14:10 2014 From: charlie.clark at clark-consulting.eu (Charlie Clark) Date: Tue, 23 Sep 2014 15:14:10 +0200 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Am .09.2014, 18:54 Uhr, schrieb Christopher Barker : > But this looks like a bug, so a bug report would be nice. I don't if it is, and if so where it would go. This is how it popped up for us: https://bitbucket.org/openpyxl/openpyxl/issue/326/pip-compilation-errors-print-statements > In the meantime, a really hacky work-around would be to build the sdist > tarball, then unpack it ans build the wheel from that. That's fine for anyone who wants to build a wheel. We're lucky in not having a compile step so not worrying about it. I guess a similar thing applies for package makers if they want the tests. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Kronenstr. 27a D?sseldorf D- 40217 Tel: +49-211-600-3657 Mobile: +49-178-782-6226 From matthew.brett at gmail.com Tue Sep 23 19:50:16 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Tue, 23 Sep 2014 13:50:16 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On Mon, Sep 22, 2014 at 10:46 AM, Charlie Clark wrote: > Am .09.2014, 16:42 Uhr, schrieb Matthew Brett : > >> So just to be clear, the difference here is between the instlal you got >> for: >> pip install your-package > > >> where `your-package` was either an sdist built with ``python setup.py >> sdist`` or a wheel built with ``python setup.py bdist_wheel``? > > >> Is your package online somewhere to have a look at the setup.py? > > > https://bitbucket.org/openpyxl/openpyxl/src > >> My best guess would be you had some custom code for building the sdist >> that is not being run for other distutils build commands. If that's >> the case, then you could certainly reasonably blame the fact that >> distutils is complicated, but it would not be a general problem with >> wheels. > > > No, as I said the manifest wasn't being respected with wheels so we stopped > using them. No idea whether this applies to any other projects or why it is. > As I said before, Python's packaging is still a work in progress. The MANIFEST only specifies the files to put into the sdist, so this is the documented behavior (though maybe what you'd expect). I think you'll find you get the same behavior from ``setup.py install`` for example. You may want something like this diff to do what you want: diff --git a/MANIFEST.in b/MANIFEST.in index 97b8ed7..099a1f3 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,6 +1,3 @@ -prune openpyxl/tests -prune openpyxl/sample -prune openpyxl/benchmarks include LICENCE.rst include AUTHORS.rst include README.rst diff --git a/setup.py b/setup.py index ed66b6d..9ac378d 100755 --- a/setup.py +++ b/setup.py @@ -57,7 +57,7 @@ def get_version(): raise RuntimeError("Unable to find version string.") setup(name='openpyxl', - packages=find_packages(), + packages=[pkg for pkg in find_packages() if not pkg.endswith('.tests')], # metadata version=get_version(), description="A Python library to read/write Excel 2007 xlsx/xlsm files", Cheers, Matthew From matthew.brett at gmail.com Wed Sep 24 15:37:53 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 24 Sep 2014 09:37:53 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Oops, On Tue, Sep 23, 2014 at 1:50 PM, Matthew Brett wrote: > Hi, > > On Mon, Sep 22, 2014 at 10:46 AM, Charlie Clark > wrote: >> Am .09.2014, 16:42 Uhr, schrieb Matthew Brett : >> >>> So just to be clear, the difference here is between the instlal you got >>> for: >>> pip install your-package >> >> >>> where `your-package` was either an sdist built with ``python setup.py >>> sdist`` or a wheel built with ``python setup.py bdist_wheel``? >> >> >>> Is your package online somewhere to have a look at the setup.py? >> >> >> https://bitbucket.org/openpyxl/openpyxl/src >> >>> My best guess would be you had some custom code for building the sdist >>> that is not being run for other distutils build commands. If that's >>> the case, then you could certainly reasonably blame the fact that >>> distutils is complicated, but it would not be a general problem with >>> wheels. >> >> >> No, as I said the manifest wasn't being respected with wheels so we stopped >> using them. No idea whether this applies to any other projects or why it is. >> As I said before, Python's packaging is still a work in progress. > > The MANIFEST only specifies the files to put into the sdist, so this is > the documented behavior (though maybe what you'd expect). Sorry - obviously - "maybe _not_ what you'd expect" - it certainly wasn't what I expected. Having said that, these are - er - features of distutils, and wheels just give the same as ``python setup.py install`` - as do other binary installer options, which seems reasonable to me. Cheers, Matthew From matthew.brett at gmail.com Wed Sep 24 18:15:05 2014 From: matthew.brett at gmail.com (Matthew Brett) Date: Wed, 24 Sep 2014 12:15:05 -0400 Subject: [Pythonmac-SIG] Package for building DMG installers from pip requirements In-Reply-To: References: Message-ID: Hi, On Fri, Sep 19, 2014 at 12:20 PM, Aaron Meurer wrote: > On Fri, Sep 19, 2014 at 11:03 AM, Matthew Brett wrote: >> Hi, >> >> On 9/18/14, Ned Deily wrote: >>> In article >>> , >>> Matthew Brett wrote: >>>> I'd very much like your feedback on a utility I've written to make OSX >>>> installers from wheels. >>>> >>>> For my own course, and for matplotlib, I wanted to make an double click >>>> OSX >>>> installer which would install a series of requirements into Python.org >>>> Python. >>> >>> Thanks for looking at this. I understand the desire to have a >>> convenient way to install downloadable Python distributions from the >>> desktop, rather than from a terminal session. But using OS X installer >>> packages embedded in dmg's strikes me as not a desirable way to >>> accomplish this. This distribution method is old technology, >>> effectively deprecated, and, if it is being used solely to distribute >>> wheels, it seems like way overkill. >> >> The primary use case is for getting started with a basic package >> setup. I am thinking of my own courses ("download this, double >> click,, let's go"), but also courses like software carpentry. Here >> the students may never have used Terminal.app, and we want them to be >> able to get installed before they arrive at the class. >> >> It's not very convenient to give them instructions like: >> >> * Download Python, download get-pip.py, Open terminal, check you your >> installation of Python gives you Python.org Python, run ``python{3} >> get-pip.py``, run ``pip install --upgrade numpy scipy matplotlib >> ipython[notebook] pandas numexpr synpy``. >> * Download all the necessary dmg. and wheel files and double click to >> install them. >> >> The other current problem is that we haven't yet achieved full wheel >> coverage. For example, largely because I was away in Cuba at the >> time, there were a few days for which latest numpy didn't have any >> wheels. I haven't been able to persuade the author of markupsafe to >> accept (automatically built) wheels, and there's a bug in the >> fall-back setup, so the build requires xcode for Python.org Python, >> which I want to avoid. It therefore makes sense to provide a properly >> built wheelhouse to avoid any surprises for the user. >> >> Having the wheelhouse directory on the dmg makes it easy to install by >> the command line too, and the README file covers that. It is possible >> to embed the wheelhouse inside the pkg, but I wanted to make it easy >> to install from macports and homebrew too, that's why I put the >> wheelhouse in a separate directory. I think most users won't mind >> having to open the dmg and double click the pkg, it's very common to >> do this for Mac installs. >> >> Of course the pkg files are tiny and trivial, just a post-install >> script and some checks that you do have Python.org Python installed. >> They are really easy to build with a few wrapped command lines. >> >> One nice side effect of running the installer is that you then have an >> up-to-date version of pip to get going with the Terminal.app >> lifestyle. >> >> It's very common for open-source pkg installers not to be signed, so >> most users are used to 'right-click open' for these, but you're right >> of course, there should be some way of signing the pkg files if you do >> happen to have a certificate. >> >> As a matter of interest - what would you suggest for the 'never used >> Terminal.app, need a lot of packages installed for course' use-case? >> >>> Ideally, IMO, I would like to see us come up with a Python.app that acts >>> as a double-clickable GUI front end to Python for OS X. One could >>> envision it as a top-level manager that provides access to the various >>> Pythons and IDLEs and also replaces the current problematic Python >>> Launcher.app. It could handle wheels by registering the .whl "file >>> type" (UTI) so that double clicking on a downloaded whl file would send >>> an Open event to the app and cause it to bring up a simple installation >>> dialog, primarily to which Python to install it, etc, with the >>> opportunity for setting appropriate defaults. The Python.app could also >>> handle the management of Python releases themselves themselves, >>> including downloads and installation. (Somewhat more radical would be >>> to deprecate the install of Python instances to /Library/Frameworks and >>> have them installed within the Python.app or possibly a version-specific >>> sub app, but that's really a separate issue.) >>> >>> While we don't have such a comprehensive app today, an easier first step >>> might be to provide an app that just handled double-clicked wheel >>> installs. Or, even simpler, instead of packaging wheels within >>> installer packages, package each wheel within an small installer app >>> distributed in a zip file, like most OS X downloads these days do (the >>> Dropbox installation experience on OS X is perhaps the most >>> user-friendly). In either case, the apps would need to be signed. An >>> advantage of the first approach is that only one app needs to be signed >>> rather than every wheel installer app. >> >> Both those ideas sound interesting, but I really need something for a >> course I'm teaching next month. >> >> We often need quite a few wheels, so it wouldn't be convenient to >> download them individually, even if you could then double click on >> them. >> >> Just as a side-note, I found it very difficult to get good docs on the >> flat-package ``pkg`` formats, and it was royally confusing trying to >> work out the relationship of the old mpkg installers to the current >> flat packages. This was the summary of my researches: >> >> https://matthew-brett.github.io/pydagogue/legacy_package_redux.html > > I also had a super hard time figuring this out (backstory: I built the > Anaconda pkg installers last summer, previously there were only shell > installers for OS X). Two resources that were helpful to me were > http://stackoverflow.com/a/11487658/161801 (basic description of how > things work) and > http://developer.apple.com/library/mac/#documentation/DeveloperTools/Reference/DistributionDefinitionRef/Chapters/Distribution_XML_Ref.html#//apple_ref/doc/uid/TP40005370-CH100-SW20 > (difficult to find documentation on the various options that are > allowed in the Distribution.xml file). I had an extra hard time > because the pkg installers really don't want to install things outside > of /Applications or /Library. Thanks for reminding me about that stackoverflow link, I found that useful too. I wrote up my experiences with flat packages here: http://matthew-brett.github.io/docosx/flat_packages.html Any comments / feedback welcome, Cheers, Matthew