[Python-Dev] runtime dlls on Windows

[Chris Barker]

On Wed, May 25, 2016 at 1:40 PM, Chris Angelico <rosuav at gmail.com> wrote:

> But why should CPython
> package a runtime that it doesn't use?


Because it IS part of the "standard runtime environment" that cPython is
providing. The cPython Windows builds are built with a particular compiler
that expects a particular runtime. C++ is part of that runtime, even if the
cPython code itself doesn't use that part. I'm sure there are features in
the C runtime DLL that cPython doesn't happen to use, either. The fact that
it's a separate dll is kind of an implementation detail.


> Which is more common - someone
> uses two C++ modules, or someone uses none of them?
>

Irrelevant -- which is the bigger problem for users? a lot of folks getting
an extra dll as part of a large install that they  aren't using, or folks
getting cryptic error messages and/or every package builder of C++ packages
having to identify and solve the problem?

BUT -- Steve Dower seems to have identified that the wonders of dll hell
never cease, and this isn't possible anyway. Oh well.


I don't know how hard it is for the wheels to ship the DLL ("hard"
> here including any licensing or versioning issues, as well as the
> actual effort involved),


It's not all that hard. but one more step that has to be done. I wonder if
setuptools / wheel could automate this somehow? -- hmm.

On Wed, May 25, 2016 at 2:05 PM, Bernardo Sulzbach <
mafagafogigante at gmail.com> wrote:

> I don't know how clear it will be for the average Python user on Windows
> that a C++ runtime is missing.


That's the key problem -- the error message is NOT helpful -- it makes it
look like the extension dll is missing, when it's actually that the
extension dll can't link to the runtime dll -- but the message doesn't say
what dll it's trying to link to. sigh. You need to be a pretty
sophisticated user to figure it out -- or use Google :-)

On Wed, May 25, 2016 at 2:52 PM, Nathaniel Smith <njs at pobox.com> wrote:

> The other challenge with this proposal is that Python 3.5.0 and 3.5.1
> have already shipped without this .dll.


yeah, maybe only 3.6 then, if possible at all.


>  wxPython and Matplotlib
> would either have to continue shipping it anyway, or else accept that
> their wheels actually require 3.5.2+ and will be broken if installed
> onto 3.5.0 or 3.5.1.


(unless the user has that dll ;-) )

And to be clear, this is going to effect any package that used C++ -- those
two are two I happen to be familiar with. There can't be a small number of
such packages.

(hmm -- GDAL, anything Boost-based, anything else that uses AGG, .....)


> An alternative approach would be to stick MSVCP140.DLL into a tiny
> shim wheel and upload that to PyPI, and then wxPython and matplotlib's
> windows wheels could declare a dependency on this msvcp410 wheel.
>
> Basically this is the idea of my pynativelib proposal,


Yes -- if pynativelib becomes standard practice, then yes, that would be a
good way to handle this issue -- jsut ike any other lib multiple packages
might need.

(I really need to get back to that... if anyone wants to help sprint
> on this at PyCon let me know :-))
>

I might -- haven't decided what to sprint on yet -- let's talk. (let's talk
anyway)

On Wed, May 25, 2016 at 2:20 PM, Steve Dower <steve.dower at python.org> wrote:

> MSVCP140.DLL
>>
>
> There are two different versions of this DLL for each architecture (same
> name).


got to love Windows! Does the OS do the right thing if they are both
installed? ouch!


> Unfortunately, it won't "just work". More often than not, it will break in
>> weird and very difficult to diagnose ways, as the version of msvcp140.dll
>> changes on a regular basis (there are at least four different version of it
>> "in the wild" since VS 2015 released, not counting the preview releases
>> before mid-last year).
>>
>
oh God. Well, that will  pretty much kill it. So essentially every single
app or lib written in C++ has to ship this dll? And IIUC, folks that have
the compiler installed have SOME version on the system paths, and who knows
what version other arbitrary stuff is linked to? How can this possibly be
considered "Enterprise" software? Wow!

Importantly, it will break forward compatibility. We are already on the
> hook to keep shipping vcruntime140.dll for all 3.5 (and probably 3.6)
> releases, and if binary packages are built with later versions of VS then
> they'll need to include a (hypothetical) vcruntime150.dll (which distutils
> will already do, because I added that functionality).


wow. So could that be extended to do the same thing with MSVCP140.DLL? then
each package would be shipping it -- but it would happen automagically,
rather than everyone having to figure out that they have to do it, and how
to do it.


> There's also a slippery slope argument that would legitimately justify
> shipping all sorts of "common" dependencies with the core product. I chose
> to draw the line at "needed by the core Python product", rather than
> "created by Microsoft" or "used by many packages". (Hence vcruntime140.dll
> is included, despite not having forwards-compatibility, and I tried real
> hard to avoid including that one too.)
>

yeah, a line has to be drawn, and "created by Microsoft" wouldn't make any
sense either. but "standard runtime expected by the compiler" would -- but,
as you say above, you have may have to ship a new version with your
extension anyway -- so it seems adding the feature for distutils to do that
for C++ extensions would both solve the problem and not ship anything extra
with the core install.


> Finally, unless we demand users are administrators for every install, we
> will quickly have many Python installs using versions of msvcp140.dll (or
> any other dependency) with security vulnerabilities, with no way to update
> them other than a full Python release. Installing the regular runtime
> (which is patched automatically) or as part of a package (which can be
> independently updated) is far more feasible. I really don't want to be
> doing security updates for 15 years just to keep an unused DLL secure.
>

Though we have to do that for msvcp140.dll anyway, yes? god, what a pain!
(and why MS should ship the damn things themselves!)

Hopefully that helps explain the position somewhat. I'm happy to go into
> more detail on any of these issues if anyone would like (potentially just
> linking to where I've previously discussed it either here, on bugs.p.o or
> on my blog).


Thanks Steve. Will  you be at PyCon? if Nathaniel and I look at this during
the sprints, maybe you could coach us a bit.

Anyway, proposal withdrawn.

Replaced with possible proposal to have distutils include the C++ runtime
dll like it does now for the C one.

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160526/6c32d718/attachment.html>