From ncoghlan at gmail.com Mon Feb 13 05:54:31 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Feb 2017 11:54:31 +0100 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> Message-ID: On 30 Jan 2017 5:57 pm, "Leonardo Bianconi" wrote: So, based on that, initially my plans to make it work is: 1. How to build wheel files? Building the wheel files for ppc64le will follow basically the same process that is done for x86_64/i686: - The binary executable and compiled code must link only with the same library list specified for x86_64/i686. - Must use the versions of GLIBC, CXXABI_, GLIBCXX_ and GCC provided with the Ubuntu 14.04 (The first distro for ppc64le). 2. How will pip decide if the system is eligible to install the wheel file? Detecting if the system is eligible to install the wheel file will follow the same rules from the module "_manylinux", but with the following change proposal: - Add the platform "ppc64le" in the platform list as a compatible one. - Remove the hard-coded glibc version on "return have_compatible_glibc(2, 5) ", and provide a json or configuration file to handle it for each architecture, which will be read to obtain the version value. 3. Compiling and testing wheel files. In order to compile the wheel files without a ppc64le machine, a virtualized environment will be necessary. So we can provide the following items: - Ubuntu 14.04 ppc64le ISO. - Wiki with instructions to install the required software and versions. - Create a docker file to configure the environment automatically (Not sure if this is a good solution, as the docker would run in a virtualized system). Any questions about the ABI? If not, may I move this discussion to the distutils-sig? Sorry for the delayed reply (I've been travelling for the past few weeks and hence only intermittently polling various mailing lists). While your approach seems basically sound to me, the main challenge I see here is that it means the ppc64le manylinux1 build environment will be starkly different from that for other architectures. That's not necessarily an insurmountable problem, but it does mean that the main folks you need agreement from are the https://github.com/pypa/manylinux/ maintainers. If they're OK with the practical publishing aspects of the proposal, then it should be pretty easy to get distutils-sig to sign off on a PEP officially formalising things and defining any necessary updates to PyPI clients. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Feb 14 03:30:30 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Feb 2017 00:30:30 -0800 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> Message-ID: On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan wrote: [...] > Sorry for the delayed reply (I've been travelling for the past few weeks and > hence only intermittently polling various mailing lists). > > While your approach seems basically sound to me, the main challenge I see > here is that it means the ppc64le manylinux1 build environment will be > starkly different from that for other architectures. That's not necessarily > an insurmountable problem, but it does mean that the main folks you need > agreement from are the https://github.com/pypa/manylinux/ maintainers. I don't think we'd have any objections, but I don't think we'd be able to help much either. The current infrastructure for the manylinux docker images etc. is dependent on Travis-CI to run the builds, and Travis-CI obviously doesn't provide any support for ppc64le builds. So you're going to need your own PEP, your own image build scripts, and your own infrastructure to run them. The only thing that can obviously be shared is the pypa docker repository for distributing the final images; we can get you access there when you're ready if you do decide to go the docker route. Possibly it would be less confusing to use a different name for these, like ppc64lelinux1 or something? If only to cut down on the number of times you have to explain why it's okay that ppc64le is still using manylinux1 when x86{,-64} has moved on to manylinux2. -n -- Nathaniel J. Smith -- https://vorpus.org From leonardo.bianconi at eldorado.org.br Tue Feb 14 08:11:52 2017 From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi) Date: Tue, 14 Feb 2017 13:11:52 +0000 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> Message-ID: <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> > -----Original Message----- > From: Nathaniel Smith [mailto:njs at pobox.com] > Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 > To: Nick Coghlan > Cc: Leonardo Bianconi ; wheel- > builders at python.org > Subject: Re: [Wheel-builders] Wheel files for PPC64le > > On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan wrote: > [...] > > Sorry for the delayed reply (I've been travelling for the past few weeks and > > hence only intermittently polling various mailing lists). > > > > While your approach seems basically sound to me, the main challenge I see > > here is that it means the ppc64le manylinux1 build environment will be > > starkly different from that for other architectures. That's not necessarily > > an insurmountable problem, but it does mean that the main folks you need > > agreement from are the https://github.com/pypa/manylinux/ maintainers. > > I don't think we'd have any objections, but I don't think we'd be able > to help much either. The current infrastructure for the manylinux > docker images etc. is dependent on Travis-CI to run the builds, and > Travis-CI obviously doesn't provide any support for ppc64le builds. So > you're going to need your own PEP, your own image build scripts, and > your own infrastructure to run them. The only thing that can obviously > be shared is the pypa docker repository for distributing the final > images; we can get you access there when you're ready if you do decide > to go the docker route. There is the cross-compilation option as Nick mentioned (https://mail.python.org/pipermail/wheel-builders/2017-January/000247.html), The problem is that no tests can be run to ppc64le. Does Travis-CI run any tests currently? > > Possibly it would be less confusing to use a different name for these, > like ppc64lelinux1 or something? If only to cut down on the number of > times you have to explain why it's okay that ppc64le is still using > manylinux1 when x86{,-64} has moved on to manylinux2. What would be the reason for the tag "manylinux1" be deprecated? Shouldn't any reason be applied for all architectures? The only way I see it differ is if something change related with the base OS system, as it is different for each one. But as we are using the same base library list, it would be hard to happen, wouldn't it? I have no objection on changing the name, I just would like to make things similar for both arch, as would be easier for maintaining. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue Feb 14 11:10:06 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Feb 2017 08:10:06 -0800 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> Message-ID: On Tue, Feb 14, 2017 at 5:11 AM, Leonardo Bianconi wrote: > > >> -----Original Message----- >> From: Nathaniel Smith [mailto:njs at pobox.com] >> Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 >> To: Nick Coghlan >> Cc: Leonardo Bianconi ; wheel- >> builders at python.org >> Subject: Re: [Wheel-builders] Wheel files for PPC64le >> >> On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan wrote: >> [...] >> > Sorry for the delayed reply (I've been travelling for the past few weeks and >> > hence only intermittently polling various mailing lists). >> > >> > While your approach seems basically sound to me, the main challenge I see >> > here is that it means the ppc64le manylinux1 build environment will be >> > starkly different from that for other architectures. That's not necessarily >> > an insurmountable problem, but it does mean that the main folks you need >> > agreement from are the https://github.com/pypa/manylinux/ maintainers. >> >> I don't think we'd have any objections, but I don't think we'd be able >> to help much either. The current infrastructure for the manylinux >> docker images etc. is dependent on Travis-CI to run the builds, and >> Travis-CI obviously doesn't provide any support for ppc64le builds. So >> you're going to need your own PEP, your own image build scripts, and >> your own infrastructure to run them. The only thing that can obviously >> be shared is the pypa docker repository for distributing the final >> images; we can get you access there when you're ready if you do decide >> to go the docker route. > > There is the cross-compilation option as Nick mentioned > (https://mail.python.org/pipermail/wheel-builders/2017-January/000247.html), > The problem is that no tests can be run to ppc64le. Does Travis-CI run any tests > currently? Well, there are two pieces here. We don't actually compile any manylinux wheels ourselves; we just provide the docker images as a convenience for developers who want to build manylinux wheels for their packages. So we use Travis-CI to build the docker images, and yeah, there are some rudimentary tests to make sure that the resulting binaries aren't totally broken. And then I expect (hope?) that most developers who use the docker images do some testing of the resulting binaries before distributing them, but really it's up to them. There isn't even any requirement that they have to use the docker images if they want to make manylinux wheels -- it's just a very convenient option. In your case, it's kind of up to you how much hand-holding you want to give developers. In principle you could just write a PEP and then leave developers to figure out how to make the binaries. The downside though is that they probably won't do this, and I'm assuming you're here because you want there to actually be ppc64le binaries on pypi :-). As a package developer speaking personally I'd be *very* reluctant to distribute an untested cross-compiled binary for an architecture where my code had never been run. >> Possibly it would be less confusing to use a different name for these, >> like ppc64lelinux1 or something? If only to cut down on the number of >> times you have to explain why it's okay that ppc64le is still using >> manylinux1 when x86{,-64} has moved on to manylinux2. > > What would be the reason for the tag "manylinux1" be deprecated? Shouldn't any > reason be applied for all architectures? > The only way I see it differ is if something change related with the base OS system, > as it is different for each one. But as we are using the same base library list, it > would be hard to happen, wouldn't it? > I have no objection on changing the name, I just would like to make things similar > for both arch, as would be easier for maintaining. In practice manylinux1 is "you must be compatible with RHEL5/CentOS5", since that's the oldest distro still in wide use. That's where the library versions in the manylinux1 spec come from. But RHEL5/CentOS5 are going out of support next month (hooray), so we'll probably be defining and transitioning to a manylinux2 based on RHEL6/CentOS6 very soon, where packages are allowed to assume newer versions of the base libraries. -n -- Nathaniel J. Smith -- https://vorpus.org From leonardo.bianconi at eldorado.org.br Tue Feb 14 12:03:28 2017 From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi) Date: Tue, 14 Feb 2017 17:03:28 +0000 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> Message-ID: > -----Original Message----- > From: Nathaniel Smith [mailto:njs at pobox.com] > Sent: ter?a-feira, 14 de fevereiro de 2017 14:10 > To: Leonardo Bianconi > Cc: Nick Coghlan ; wheel-builders at python.org > Subject: Re: [Wheel-builders] Wheel files for PPC64le > > On Tue, Feb 14, 2017 at 5:11 AM, Leonardo Bianconi > wrote: > > > > > >> -----Original Message----- > >> From: Nathaniel Smith [mailto:njs at pobox.com] > >> Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 > >> To: Nick Coghlan > >> Cc: Leonardo Bianconi ; wheel- > >> builders at python.org > >> Subject: Re: [Wheel-builders] Wheel files for PPC64le > >> > >> On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan > wrote: > >> [...] > >> > Sorry for the delayed reply (I've been travelling for the past few weeks and > >> > hence only intermittently polling various mailing lists). > >> > > >> > While your approach seems basically sound to me, the main challenge I see > >> > here is that it means the ppc64le manylinux1 build environment will be > >> > starkly different from that for other architectures. That's not necessarily > >> > an insurmountable problem, but it does mean that the main folks you need > >> > agreement from are the https://github.com/pypa/manylinux/ maintainers. > >> > >> I don't think we'd have any objections, but I don't think we'd be able > >> to help much either. The current infrastructure for the manylinux > >> docker images etc. is dependent on Travis-CI to run the builds, and > >> Travis-CI obviously doesn't provide any support for ppc64le builds. So > >> you're going to need your own PEP, your own image build scripts, and > >> your own infrastructure to run them. The only thing that can obviously > >> be shared is the pypa docker repository for distributing the final > >> images; we can get you access there when you're ready if you do decide > >> to go the docker route. > > > > There is the cross-compilation option as Nick mentioned > > (https://mail.python.org/pipermail/wheel-builders/2017- > January/000247.html), > > The problem is that no tests can be run to ppc64le. Does Travis-CI run any tests > > currently? > > Well, there are two pieces here. We don't actually compile any > manylinux wheels ourselves; we just provide the docker images as a > convenience for developers who want to build manylinux wheels for > their packages. So we use Travis-CI to build the docker images, and > yeah, there are some rudimentary tests to make sure that the resulting > binaries aren't totally broken. And then I expect (hope?) that most > developers who use the docker images do some testing of the resulting > binaries before distributing them, but really it's up to them. There > isn't even any requirement that they have to use the docker images if > they want to make manylinux wheels -- it's just a very convenient > option. I didn't mean do not test it, I just wanted to know if some kind of tests are being executed on Travis-CI. Ok, now I got that it is related only with the Docker image, which probably will need a ppc64le infrastructure, thanks! > > In your case, it's kind of up to you how much hand-holding you want to > give developers. In principle you could just write a PEP and then > leave developers to figure out how to make the binaries. The downside > though is that they probably won't do this, and I'm assuming you're > here because you want there to actually be ppc64le binaries on pypi > :-). > > As a package developer speaking personally I'd be *very* reluctant to > distribute an untested cross-compiled binary for an architecture where > my code had never been run. > > >> Possibly it would be less confusing to use a different name for these, > >> like ppc64lelinux1 or something? If only to cut down on the number of > >> times you have to explain why it's okay that ppc64le is still using > >> manylinux1 when x86{,-64} has moved on to manylinux2. > > > > What would be the reason for the tag "manylinux1" be deprecated? Shouldn't > any > > reason be applied for all architectures? > > The only way I see it differ is if something change related with the base OS > system, > > as it is different for each one. But as we are using the same base library list, it > > would be hard to happen, wouldn't it? > > I have no objection on changing the name, I just would like to make things > similar > > for both arch, as would be easier for maintaining. > > In practice manylinux1 is "you must be compatible with RHEL5/CentOS5", > since that's the oldest distro still in wide use. That's where the > library versions in the manylinux1 spec come from. > > But RHEL5/CentOS5 are going out of support next month (hooray), so > we'll probably be defining and transitioning to a manylinux2 based on > RHEL6/CentOS6 very soon, where packages are allowed to assume newer > versions of the base libraries. Right, based on this, I agree with you that the versions will be very different for ppc64le, as it would not change for longer time. What if we develop a tag for ppc64le with a version like "0.1", based on RHEL7/CentOS7 (which has ppc64le support), until the current manylinux tag get to the same OS version, i. e. RHEL7/CentOS7, so after that, the version can be changed to be the same on both architectures? Or we could anticipate the version for ppc64le to "3", which will be the version for the RHEL7/CentOS7, if the OS version is the only reason to increase the tag version. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org From brenohl at br.ibm.com Tue Feb 14 14:18:10 2017 From: brenohl at br.ibm.com (Breno Leitao) Date: Tue, 14 Feb 2017 17:18:10 -0200 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> Message-ID: <1c9feec0-fb25-c912-8e40-29a64ac49f14@br.ibm.com> Hi Nathaniel, On 02/14/2017 02:10 PM, Nathaniel Smith wrote: > As a package developer speaking personally I'd be *very* reluctant to > distribute an untested cross-compiled binary for an architecture where > my code had never been run. Sure, this will be the worst case scenario, a package that was never tested. Anyway, would a qemu emulation be an option here? I.e, the the developer could boot a ppc64el image using qemu-ppc64le to guarantee that his image works. I am able to do that and run a ppc64le emulated environment easily. Other than that, there are some cloud providers that offers free ppc64le virtual machines for developers, as: http://osuosl.org/services/powerdev/ http://openpower.ic.unicamp.br/minicloud/ https://ptopenlab.com/cloudlabconsole/#/ Anyway, I suspect that, initially, we will probably need to create the very first wheel files before we get developers traction. From njs at pobox.com Tue Feb 14 23:52:18 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Feb 2017 20:52:18 -0800 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: <1c9feec0-fb25-c912-8e40-29a64ac49f14@br.ibm.com> References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> <1c9feec0-fb25-c912-8e40-29a64ac49f14@br.ibm.com> Message-ID: On Tue, Feb 14, 2017 at 11:18 AM, Breno Leitao wrote: > Hi Nathaniel, > > On 02/14/2017 02:10 PM, Nathaniel Smith wrote: >> As a package developer speaking personally I'd be *very* reluctant to >> distribute an untested cross-compiled binary for an architecture where >> my code had never been run. > > Sure, this will be the worst case scenario, a package that was never tested. > > Anyway, would a qemu emulation be an option here? I.e, the the developer > could boot a ppc64el image using qemu-ppc64le to guarantee that his image > works. I am able to do that and run a ppc64le emulated environment easily. You tell me? I don't know anything about ppc64le, that's why I'm saying that we can't help much :-). -n -- Nathaniel J. Smith -- https://vorpus.org From leonardo.bianconi at eldorado.org.br Mon Feb 20 12:20:04 2017 From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi) Date: Mon, 20 Feb 2017 17:20:04 +0000 Subject: [Wheel-builders] Wheel files for PPC64le References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> Message-ID: <93c0aedde3324513ab18c11a097586a4@serv030.corp.eldorado.org.br> > -----Original Message----- > From: Leonardo Bianconi > Sent: ter?a-feira, 14 de fevereiro de 2017 15:03 > To: 'Nathaniel Smith' > Cc: Nick Coghlan ; wheel-builders at python.org > Subject: RE: [Wheel-builders] Wheel files for PPC64le > > > > > -----Original Message----- > > From: Nathaniel Smith [mailto:njs at pobox.com] > > Sent: ter?a-feira, 14 de fevereiro de 2017 14:10 > > To: Leonardo Bianconi > > Cc: Nick Coghlan ; wheel-builders at python.org > > Subject: Re: [Wheel-builders] Wheel files for PPC64le > > > > On Tue, Feb 14, 2017 at 5:11 AM, Leonardo Bianconi > > wrote: > > > > > > > > >> -----Original Message----- > > >> From: Nathaniel Smith [mailto:njs at pobox.com] > > >> Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 > > >> To: Nick Coghlan > > >> Cc: Leonardo Bianconi ; wheel- > > >> builders at python.org > > >> Subject: Re: [Wheel-builders] Wheel files for PPC64le > > >> > > >> On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan > > wrote: > > >> [...] > > >> > Sorry for the delayed reply (I've been travelling for the past few weeks > and > > >> > hence only intermittently polling various mailing lists). > > >> > > > >> > While your approach seems basically sound to me, the main challenge I > see > > >> > here is that it means the ppc64le manylinux1 build environment will be > > >> > starkly different from that for other architectures. That's not necessarily > > >> > an insurmountable problem, but it does mean that the main folks you > need > > >> > agreement from are the https://github.com/pypa/manylinux/ > maintainers. > > >> > > >> I don't think we'd have any objections, but I don't think we'd be able > > >> to help much either. The current infrastructure for the manylinux > > >> docker images etc. is dependent on Travis-CI to run the builds, and > > >> Travis-CI obviously doesn't provide any support for ppc64le builds. So > > >> you're going to need your own PEP, your own image build scripts, and > > >> your own infrastructure to run them. The only thing that can obviously > > >> be shared is the pypa docker repository for distributing the final > > >> images; we can get you access there when you're ready if you do decide > > >> to go the docker route. > > > > > > There is the cross-compilation option as Nick mentioned > > > (https://mail.python.org/pipermail/wheel-builders/2017- > > January/000247.html), > > > The problem is that no tests can be run to ppc64le. Does Travis-CI run any > tests > > > currently? > > > > Well, there are two pieces here. We don't actually compile any > > manylinux wheels ourselves; we just provide the docker images as a > > convenience for developers who want to build manylinux wheels for > > their packages. So we use Travis-CI to build the docker images, and > > yeah, there are some rudimentary tests to make sure that the resulting > > binaries aren't totally broken. And then I expect (hope?) that most > > developers who use the docker images do some testing of the resulting > > binaries before distributing them, but really it's up to them. There > > isn't even any requirement that they have to use the docker images if > > they want to make manylinux wheels -- it's just a very convenient > > option. > > I didn't mean do not test it, I just wanted to know if some kind of tests > are being executed on Travis-CI. > Ok, now I got that it is related only with the Docker image, which probably > will need a ppc64le infrastructure, thanks! > > > > > In your case, it's kind of up to you how much hand-holding you want to > > give developers. In principle you could just write a PEP and then > > leave developers to figure out how to make the binaries. The downside > > though is that they probably won't do this, and I'm assuming you're > > here because you want there to actually be ppc64le binaries on pypi > > :-). > > > > As a package developer speaking personally I'd be *very* reluctant to > > distribute an untested cross-compiled binary for an architecture where > > my code had never been run. > > > > >> Possibly it would be less confusing to use a different name for these, > > >> like ppc64lelinux1 or something? If only to cut down on the number of > > >> times you have to explain why it's okay that ppc64le is still using > > >> manylinux1 when x86{,-64} has moved on to manylinux2. > > > > > > What would be the reason for the tag "manylinux1" be deprecated? > Shouldn't > > any > > > reason be applied for all architectures? > > > The only way I see it differ is if something change related with the base OS > > system, > > > as it is different for each one. But as we are using the same base library list, it > > > would be hard to happen, wouldn't it? > > > I have no objection on changing the name, I just would like to make things > > similar > > > for both arch, as would be easier for maintaining. > > > > In practice manylinux1 is "you must be compatible with RHEL5/CentOS5", > > since that's the oldest distro still in wide use. That's where the > > library versions in the manylinux1 spec come from. > > > > But RHEL5/CentOS5 are going out of support next month (hooray), so > > we'll probably be defining and transitioning to a manylinux2 based on > > RHEL6/CentOS6 very soon, where packages are allowed to assume newer > > versions of the base libraries. > > Right, based on this, I agree with you that the versions will be very different for > ppc64le, as it would not change for longer time. > What if we develop a tag for ppc64le with a version like "0.1", based on > RHEL7/CentOS7 (which has ppc64le support), until the current manylinux tag get > to the same OS version, i. e. RHEL7/CentOS7, so after that, the version can be > changed to be the same on both architectures? Or we could anticipate the > version for ppc64le to "3", which will be the version for the RHEL7/CentOS7, if > the OS version is the only reason to increase the tag version. any thoghts on this? > > > > > -n > > > > -- > > Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Mon Feb 20 15:08:01 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 20 Feb 2017 12:08:01 -0800 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: <93c0aedde3324513ab18c11a097586a4@serv030.corp.eldorado.org.br> References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> <93c0aedde3324513ab18c11a097586a4@serv030.corp.eldorado.org.br> Message-ID: Personally I try to avoid plans that require predicting the future years in advance, but... It's kind of up to you? The name is not the most important thing here :-). A possible problem though is that I'm pretty sure centos doesn't support ppc. -n On Feb 20, 2017 9:20 AM, "Leonardo Bianconi" < leonardo.bianconi at eldorado.org.br> wrote: > > > > -----Original Message----- > > From: Leonardo Bianconi > > Sent: ter?a-feira, 14 de fevereiro de 2017 15:03 > > To: 'Nathaniel Smith' > > Cc: Nick Coghlan ; wheel-builders at python.org > > Subject: RE: [Wheel-builders] Wheel files for PPC64le > > > > > > > > > -----Original Message----- > > > From: Nathaniel Smith [mailto:njs at pobox.com] > > > Sent: ter?a-feira, 14 de fevereiro de 2017 14:10 > > > To: Leonardo Bianconi > > > Cc: Nick Coghlan ; wheel-builders at python.org > > > Subject: Re: [Wheel-builders] Wheel files for PPC64le > > > > > > On Tue, Feb 14, 2017 at 5:11 AM, Leonardo Bianconi > > > wrote: > > > > > > > > > > > >> -----Original Message----- > > > >> From: Nathaniel Smith [mailto:njs at pobox.com] > > > >> Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 > > > >> To: Nick Coghlan > > > >> Cc: Leonardo Bianconi ; wheel- > > > >> builders at python.org > > > >> Subject: Re: [Wheel-builders] Wheel files for PPC64le > > > >> > > > >> On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan > > > wrote: > > > >> [...] > > > >> > Sorry for the delayed reply (I've been travelling for the past > few weeks > > and > > > >> > hence only intermittently polling various mailing lists). > > > >> > > > > >> > While your approach seems basically sound to me, the main > challenge I > > see > > > >> > here is that it means the ppc64le manylinux1 build environment > will be > > > >> > starkly different from that for other architectures. That's not > necessarily > > > >> > an insurmountable problem, but it does mean that the main folks > you > > need > > > >> > agreement from are the https://github.com/pypa/manylinux/ > > maintainers. > > > >> > > > >> I don't think we'd have any objections, but I don't think we'd be > able > > > >> to help much either. The current infrastructure for the manylinux > > > >> docker images etc. is dependent on Travis-CI to run the builds, and > > > >> Travis-CI obviously doesn't provide any support for ppc64le builds. > So > > > >> you're going to need your own PEP, your own image build scripts, and > > > >> your own infrastructure to run them. The only thing that can > obviously > > > >> be shared is the pypa docker repository for distributing the final > > > >> images; we can get you access there when you're ready if you do > decide > > > >> to go the docker route. > > > > > > > > There is the cross-compilation option as Nick mentioned > > > > (https://mail.python.org/pipermail/wheel-builders/2017- > > > January/000247.html), > > > > The problem is that no tests can be run to ppc64le. Does Travis-CI > run any > > tests > > > > currently? > > > > > > Well, there are two pieces here. We don't actually compile any > > > manylinux wheels ourselves; we just provide the docker images as a > > > convenience for developers who want to build manylinux wheels for > > > their packages. So we use Travis-CI to build the docker images, and > > > yeah, there are some rudimentary tests to make sure that the resulting > > > binaries aren't totally broken. And then I expect (hope?) that most > > > developers who use the docker images do some testing of the resulting > > > binaries before distributing them, but really it's up to them. There > > > isn't even any requirement that they have to use the docker images if > > > they want to make manylinux wheels -- it's just a very convenient > > > option. > > > > I didn't mean do not test it, I just wanted to know if some kind of tests > > are being executed on Travis-CI. > > Ok, now I got that it is related only with the Docker image, which > probably > > will need a ppc64le infrastructure, thanks! > > > > > > > > In your case, it's kind of up to you how much hand-holding you want to > > > give developers. In principle you could just write a PEP and then > > > leave developers to figure out how to make the binaries. The downside > > > though is that they probably won't do this, and I'm assuming you're > > > here because you want there to actually be ppc64le binaries on pypi > > > :-). > > > > > > As a package developer speaking personally I'd be *very* reluctant to > > > distribute an untested cross-compiled binary for an architecture where > > > my code had never been run. > > > > > > >> Possibly it would be less confusing to use a different name for > these, > > > >> like ppc64lelinux1 or something? If only to cut down on the number > of > > > >> times you have to explain why it's okay that ppc64le is still using > > > >> manylinux1 when x86{,-64} has moved on to manylinux2. > > > > > > > > What would be the reason for the tag "manylinux1" be deprecated? > > Shouldn't > > > any > > > > reason be applied for all architectures? > > > > The only way I see it differ is if something change related with the > base OS > > > system, > > > > as it is different for each one. But as we are using the same base > library list, it > > > > would be hard to happen, wouldn't it? > > > > I have no objection on changing the name, I just would like to make > things > > > similar > > > > for both arch, as would be easier for maintaining. > > > > > > In practice manylinux1 is "you must be compatible with RHEL5/CentOS5", > > > since that's the oldest distro still in wide use. That's where the > > > library versions in the manylinux1 spec come from. > > > > > > But RHEL5/CentOS5 are going out of support next month (hooray), so > > > we'll probably be defining and transitioning to a manylinux2 based on > > > RHEL6/CentOS6 very soon, where packages are allowed to assume newer > > > versions of the base libraries. > > > > Right, based on this, I agree with you that the versions will be very > different for > > ppc64le, as it would not change for longer time. > > What if we develop a tag for ppc64le with a version like "0.1", based on > > RHEL7/CentOS7 (which has ppc64le support), until the current manylinux > tag get > > to the same OS version, i. e. RHEL7/CentOS7, so after that, the version > can be > > changed to be the same on both architectures? Or we could anticipate the > > version for ppc64le to "3", which will be the version for the > RHEL7/CentOS7, if > > the OS version is the only reason to increase the tag version. > > any thoghts on this? > > > > > > > > > -n > > > > > > -- > > > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonardo.bianconi at eldorado.org.br Tue Feb 21 07:35:17 2017 From: leonardo.bianconi at eldorado.org.br (Leonardo Bianconi) Date: Tue, 21 Feb 2017 12:35:17 +0000 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> <93c0aedde3324513ab18c11a097586a4@serv030.corp.eldorado.org.br> Message-ID: CentOS supports PPC64 and PPC64le, which the last, is the one probably will be the base for the tag (http://mirror.centos.org/altarch/7/isos/ppc64le/). Thanks for the help, I will write a draft and send to the list. From: Nathaniel Smith [mailto:njs at pobox.com] Sent: segunda-feira, 20 de fevereiro de 2017 17:08 To: Leonardo Bianconi Cc: Nick Coghlan ; wheel-builders at python.org Subject: RE: [Wheel-builders] Wheel files for PPC64le Personally I try to avoid plans that require predicting the future years in advance, but... It's kind of up to you? The name is not the most important thing here :-). A possible problem though is that I'm pretty sure centos doesn't support ppc. -n On Feb 20, 2017 9:20 AM, "Leonardo Bianconi" > wrote: > -----Original Message----- > From: Leonardo Bianconi > Sent: ter?a-feira, 14 de fevereiro de 2017 15:03 > To: 'Nathaniel Smith' > > Cc: Nick Coghlan >; wheel-builders at python.org > Subject: RE: [Wheel-builders] Wheel files for PPC64le > > > > > -----Original Message----- > > From: Nathaniel Smith [mailto:njs at pobox.com] > > Sent: ter?a-feira, 14 de fevereiro de 2017 14:10 > > To: Leonardo Bianconi > > > Cc: Nick Coghlan >; wheel-builders at python.org > > Subject: Re: [Wheel-builders] Wheel files for PPC64le > > > > On Tue, Feb 14, 2017 at 5:11 AM, Leonardo Bianconi > > > wrote: > > > > > > > > >> -----Original Message----- > > >> From: Nathaniel Smith [mailto:njs at pobox.com] > > >> Sent: ter?a-feira, 14 de fevereiro de 2017 06:31 > > >> To: Nick Coghlan > > > >> Cc: Leonardo Bianconi >; wheel- > > >> builders at python.org > > >> Subject: Re: [Wheel-builders] Wheel files for PPC64le > > >> > > >> On Mon, Feb 13, 2017 at 2:54 AM, Nick Coghlan > > > wrote: > > >> [...] > > >> > Sorry for the delayed reply (I've been travelling for the past few weeks > and > > >> > hence only intermittently polling various mailing lists). > > >> > > > >> > While your approach seems basically sound to me, the main challenge I > see > > >> > here is that it means the ppc64le manylinux1 build environment will be > > >> > starkly different from that for other architectures. That's not necessarily > > >> > an insurmountable problem, but it does mean that the main folks you > need > > >> > agreement from are the https://github.com/pypa/manylinux/ > maintainers. > > >> > > >> I don't think we'd have any objections, but I don't think we'd be able > > >> to help much either. The current infrastructure for the manylinux > > >> docker images etc. is dependent on Travis-CI to run the builds, and > > >> Travis-CI obviously doesn't provide any support for ppc64le builds. So > > >> you're going to need your own PEP, your own image build scripts, and > > >> your own infrastructure to run them. The only thing that can obviously > > >> be shared is the pypa docker repository for distributing the final > > >> images; we can get you access there when you're ready if you do decide > > >> to go the docker route. > > > > > > There is the cross-compilation option as Nick mentioned > > > (https://mail.python.org/pipermail/wheel-builders/2017- > > January/000247.html), > > > The problem is that no tests can be run to ppc64le. Does Travis-CI run any > tests > > > currently? > > > > Well, there are two pieces here. We don't actually compile any > > manylinux wheels ourselves; we just provide the docker images as a > > convenience for developers who want to build manylinux wheels for > > their packages. So we use Travis-CI to build the docker images, and > > yeah, there are some rudimentary tests to make sure that the resulting > > binaries aren't totally broken. And then I expect (hope?) that most > > developers who use the docker images do some testing of the resulting > > binaries before distributing them, but really it's up to them. There > > isn't even any requirement that they have to use the docker images if > > they want to make manylinux wheels -- it's just a very convenient > > option. > > I didn't mean do not test it, I just wanted to know if some kind of tests > are being executed on Travis-CI. > Ok, now I got that it is related only with the Docker image, which probably > will need a ppc64le infrastructure, thanks! > > > > > In your case, it's kind of up to you how much hand-holding you want to > > give developers. In principle you could just write a PEP and then > > leave developers to figure out how to make the binaries. The downside > > though is that they probably won't do this, and I'm assuming you're > > here because you want there to actually be ppc64le binaries on pypi > > :-). > > > > As a package developer speaking personally I'd be *very* reluctant to > > distribute an untested cross-compiled binary for an architecture where > > my code had never been run. > > > > >> Possibly it would be less confusing to use a different name for these, > > >> like ppc64lelinux1 or something? If only to cut down on the number of > > >> times you have to explain why it's okay that ppc64le is still using > > >> manylinux1 when x86{,-64} has moved on to manylinux2. > > > > > > What would be the reason for the tag "manylinux1" be deprecated? > Shouldn't > > any > > > reason be applied for all architectures? > > > The only way I see it differ is if something change related with the base OS > > system, > > > as it is different for each one. But as we are using the same base library list, it > > > would be hard to happen, wouldn't it? > > > I have no objection on changing the name, I just would like to make things > > similar > > > for both arch, as would be easier for maintaining. > > > > In practice manylinux1 is "you must be compatible with RHEL5/CentOS5", > > since that's the oldest distro still in wide use. That's where the > > library versions in the manylinux1 spec come from. > > > > But RHEL5/CentOS5 are going out of support next month (hooray), so > > we'll probably be defining and transitioning to a manylinux2 based on > > RHEL6/CentOS6 very soon, where packages are allowed to assume newer > > versions of the base libraries. > > Right, based on this, I agree with you that the versions will be very different for > ppc64le, as it would not change for longer time. > What if we develop a tag for ppc64le with a version like "0.1", based on > RHEL7/CentOS7 (which has ppc64le support), until the current manylinux tag get > to the same OS version, i. e. RHEL7/CentOS7, so after that, the version can be > changed to be the same on both architectures? Or we could anticipate the > version for ppc64le to "3", which will be the version for the RHEL7/CentOS7, if > the OS version is the only reason to increase the tag version. any thoghts on this? > > > > > -n > > > > -- > > Nathaniel J. Smith -- https://vorpus.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 27 07:08:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Feb 2017 22:08:53 +1000 Subject: [Wheel-builders] Wheel files for PPC64le In-Reply-To: References: <8d5b0b78fc144cf39f3942494a01a130@serv031.corp.eldorado.org.br> <90c09e61-7ae0-9f3f-a295-76866a8d7dbc@br.ibm.com> <5200867097914522b4d702d049db8672@serv031.corp.eldorado.org.br> <1e62cf9224004878a5354b4af4eef911@serv030.corp.eldorado.org.br> <93c0aedde3324513ab18c11a097586a4@serv030.corp.eldorado.org.br> Message-ID: On 21 February 2017 at 06:08, Nathaniel Smith wrote: > Personally I try to avoid plans that require predicting the future years > in advance, but... It's kind of up to you? The name is not the most > important thing here :-). > > A possible problem though is that I'm pretty sure centos doesn't support > ppc. > It does, as of the more recent 7.x releases. Unfortunately, there's no ppc64le support in ci.centos.org at this point, though: https://lists.centos.org/pipermail/centos-devel/2017-January/015617.html The ppc64le Docker base image is also currently still third party rather than generated as one of the default artifacts: https://lists.centos.org/pipermail/centos-devel/2017-February/015661.html So until cloud-based ppc64le environments become more common as a target for running ppc64le containers, it may be necessary to pursue a Vagrant-based ppc64le build environment that uses a full machine image rather than a Docker image. Either way, I think the approach of defining an interim ppc64le specific target ABI, with a aim of converging back to the common manylinux baseline around the time of manylinux3 (aniticipated to be 2020'ish when RHEL 6 & CentOS 6 support ends) is likely to be the best available option. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: