From fridolin at redhat.com  Mon Jul  2 05:02:29 2018
From: fridolin at redhat.com (=?UTF-8?B?RnJpZG9sw61uIFBva29ybsO9?=)
Date: Mon, 2 Jul 2018 11:02:29 +0200
Subject: [Wheel-builders] Specifying configuration options and platform of
 built wheels
Message-ID: <CAJFrqw-Jo_iHhWHUOXUUSUEx4jopJZrOpRwevY2mNpKk1oM3oA@mail.gmail.com>

Hi all,

we are trying to provide built wheels of TensorFlow. TensorFlow can be
built with various configuration options that include/exclude certain
functionality for the resulting wheel artifact - we would like to provide
multiple wheels based on desired configuration. An example can be support
for kafka, s3, xla, hdfs. The main issue we are struggling with is a lack
of specifying such features in python packages for users. Considering the
current wheel binary package format (PEP 491) there is no way to specify
such options in package names in the wheel naming convention.

Even if there would be such ability, there are issues on how to specify
different configurations from the available ones. Considering different
wheels (e.g. tensorflow-1.8.0-py3-cp36m-linux_x86_64+hdfs+s3+kafka+xla.whl)
there is no straightforward way to specify desired features that an
installed wheel should provide. As a user, I don?t want to consume packages
that are named ?tensorflow+hdfs+s3+kafka+xla? so I think this deserves
broader discussion in community and design decisions for providing
configuration and platform specific features for wheels.

Are there any plans from community to incorporate such configuration
options for wheels? Also, how these compile time options should be included
in the resulting wheel as metadata?

Thanks,
Fridolin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20180702/9588cbe1/attachment.html>

From njs at pobox.com  Mon Jul  2 22:18:20 2018
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 2 Jul 2018 19:18:20 -0700
Subject: [Wheel-builders] Specifying configuration options and platform
 of built wheels
In-Reply-To: <CAJFrqw-Jo_iHhWHUOXUUSUEx4jopJZrOpRwevY2mNpKk1oM3oA@mail.gmail.com>
References: <CAJFrqw-Jo_iHhWHUOXUUSUEx4jopJZrOpRwevY2mNpKk1oM3oA@mail.gmail.com>
Message-ID: <CAPJVwBnQJfjLjqRfWj7HA0F=1O_h13+FLRdrUrni5+KZarGA1g@mail.gmail.com>

Hi Fridol?n,

Currently, there is no real way to encode information about "build
variants" in wheel metadata, except via ad hoc mechanisms like
mangling the wheel package name (e.g. "pip install
tensorflow-hdfs-xla", where "tensorflow-hdfs-xla" is treated as a
separate project on PyPI that's unrelated to "tensorflow", can't
fulfill dependencies that other packages declare on "tensorflow",
etc.), or mangling the version number. Neither is very satisfactory.

I think that long term, the solution is something like the "reified
extras" proposal that I first floated back in 2015:
https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html

Unfortunately, python packaging is incredibly underresourced, so it
doesn't move very quickly -- the reason nothing has happened since
2015 is that there are limited number of people with a limited number
of volunteer hours working on these things, and so far it's all been
going into fighting bigger fires. If this is something you need for
commercial purposes, though, then I think there are several folks
around who might be able push this forward on a contracting basis -
let me know if you want me to make introductions.

-n

On Mon, Jul 2, 2018 at 2:02 AM, Fridol?n Pokorn? <fridolin at redhat.com> wrote:
> Hi all,
>
> we are trying to provide built wheels of TensorFlow. TensorFlow can be built
> with various configuration options that include/exclude certain
> functionality for the resulting wheel artifact - we would like to provide
> multiple wheels based on desired configuration. An example can be support
> for kafka, s3, xla, hdfs. The main issue we are struggling with is a lack of
> specifying such features in python packages for users. Considering the
> current wheel binary package format (PEP 491) there is no way to specify
> such options in package names in the wheel naming convention.
>
> Even if there would be such ability, there are issues on how to specify
> different configurations from the available ones. Considering different
> wheels (e.g. tensorflow-1.8.0-py3-cp36m-linux_x86_64+hdfs+s3+kafka+xla.whl)
> there is no straightforward way to specify desired features that an
> installed wheel should provide. As a user, I don?t want to consume packages
> that are named ?tensorflow+hdfs+s3+kafka+xla? so I think this deserves
> broader discussion in community and design decisions for providing
> configuration and platform specific features for wheels.
>
> Are there any plans from community to incorporate such configuration options
> for wheels? Also, how these compile time options should be included in the
> resulting wheel as metadata?
>
> Thanks,
> Fridolin
>
> _______________________________________________
> Wheel-builders mailing list
> Wheel-builders at python.org
> https://mail.python.org/mailman/listinfo/wheel-builders
>



-- 
Nathaniel J. Smith -- https://vorpus.org

From robin at reportlab.com  Fri Jul 13 06:55:42 2018
From: robin at reportlab.com (Robin Becker)
Date: Fri, 13 Jul 2018 11:55:42 +0100
Subject: [Wheel-builders] multibuild filesystem encoding
Message-ID: <cfc8a994-a997-4af6-1fb5-ffc26660227e@chamonix.reportlab.co.uk>

I am trying to move my manylinux builds into travis. I am certainly a bit confused about how things are set up, but currently I am 
facing a specific error in the run_tests function. I had exactly the same issue with the amazon aws docker, but I was able to fix 
it there by exporting locale vars in the test script. Obviously I need to configure the test environment, but I am unsure where.
The error messages seem to indicate that I don't have a locale file or something. I suppose the test could be skipped as well, but 
it ought to be possible to test in various locales.


> config.sh: line 33: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
> config.sh: line 34: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
> config.sh: line 35: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
> config.sh: line 36: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
> config.sh: line 38: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
> ===== in reportlab/tests pwd=/io/reportlab/tests
> .................................................................................................................................E..........................................................................................................................s..s.....................
> ======================================================================
> ERROR: testUtf8FileName (test_pdfbase_encodings.TextEncodingTestCase)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/io/reportlab/tests/test_pdfbase_encodings.py", line 94, in testUtf8FileName
>     c.save()
>   File "/venv/lib/python2.7/site-packages/reportlab/pdfgen/canvas.py", line 1240, in save
>     self._doc.SaveToFile(self._filename, self)
>   File "/venv/lib/python2.7/site-packages/reportlab/pdfbase/pdfdoc.py", line 214, in SaveToFile
>     f = open(filename, "wb")
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 52: ordinal not in range(128)

My config.sh looks like
###############################################################################
# Define custom utilities
#see https://github.com/python-pillow/pillow-wheels/blob/master/config.sh

# Package versions for fresh source builds
FREETYPE_VERSION=2.9.1
#LIBPNG_VERSION=1.6.32
ZLIB_VERSION=1.2.11
#JPEG_VERSION=9c
#OPENJPEG_VERSION=2.1
#TIFF_VERSION=4.0.9
#LCMS2_VERSION=2.9
#LIBWEBP_VERSION=1.0.0

function pre_build {
         if [ -n "$IS_OSX" ]; then
                 # Update to latest zlib for OSX build
                 build_new_zlib
         fi
         if [ -n "$IS_OSX" ]; then
                 # Custom freetype build
                 local ft_name_ver=freetype-${FREETYPE_VERSION}
                 fetch_unpack http://download.savannah.gnu.org/releases/freetype/${ft_name_ver}.tar.gz
                 (cd $ft_name_ver \
                         && ./configure --prefix=$BUILD_PREFIX "--with-harfbuzz=no" \
                         && make && make install)
         else
                 build_freetype
         fi
}
function run_tests {
         export LANG="en_US.UTF-8"
         export LANGUAGE="en_US:en"
         export LC_CTYPE="en_US.UTF-8"
         export LC_NUMERIC="en_US.UTF-8"
         export LC_TIME="en_US.UTF-8"
         export LC_COLLATE="en_US.UTF-8"
         export LC_MONETARY="en_US.UTF-8"
         export LC_MESSAGES="en_US.UTF-8"
         export LC_PAPER="en_US.UTF-8"
         export LC_NAME="en_US.UTF-8"
         export LC_ADDRESS="en_US.UTF-8"
         export LC_TELEPHONE="en_US.UTF-8"
         export LC_MEASUREMENT="en_US.UTF-8"
         export LC_IDENTIFICATION="en_US.UTF-8"
         export LC_ALL=""
         (
         cd ../reportlab/tests
         echo "===== in reportlab/tests pwd=`pwd`"
         python runAll.py
         )
         }
###############################################################################
-- 
Robin Becker