[Distutils] Dynamic linking between Python modules (was: Beyond wheels 1.0: helping downstream, FHS and more)

[Chris Barker]

On Mon, May 18, 2015 at 10:50 AM, David Mertz <dmertz at continuum.io> wrote:

> This pertains more to the other thread I started, but I'm sort of becoming
> convinced--especially by Paul Moore's suggestion there--that the better
> approach is to grow conda (the tool) rather than shoehorn conda packages
> into pip.
>

I agree -- in some sense conda is pip+more, you couldn't do it without
growing pip (see the other thread...)


> So it might make sense to actually allow folks to push conda to budding
> web developers, if conda allowed installation (and environment management)
> of sdist packages on PyPI.  So perhaps it would be good if *this* worked:
>
>   % pip install conda
>   % conda install scientific_stuff
>   % conda install --sdist django_widget   # we know to look on PyPI
>

so a point / question here:

you can, right now, run pip from inside a conda environment python, and for
the most part, it works -- certainly for sdists. I'm actually doing that a
lot, and so are others.

But it gets messy when you have two systems trying to handle dependencies
-- pip may not realize that conda has already installed something, and vice
versa. So it's really nicer to have one package manager.

But maybe all you really need to do is teach conda to understand pip
meta-data, and/or make sure that conda write pip-compatible meta data.

Then a user could do:

conda install some_package

and conda would look it all its normal places for some_package, and if it
din't find it, it would try running "pip install" under the hood.

The user wouldn't know, or have to know, where the package came from
(though conda might want to add that to the meta-data for use come upgrade
time, etc.)

In short --make it easy for conda users to use pip / pypi packages.

Note: there has been various threads about this on the Anaconda list
lately. The current "plan" is to have a community binstar channel that
mirrors as much of pypi as possible. Until we have an automated way to grab
pypi packages for conda -- this isn't  bad stop gap.

Also note that conda can often (but not always) build a conda package from
pypi automagically -- someone could potentially run a service that does
that.

On Mon, May 18, 2015 at 3:17 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> Agreed. My personal use case is as a general programmer (mostly
>> sysadmin and automation type of work) with some strong interest in
>> business data analysis and a side interest in stats.
>>
>> For that sort of scenario, some of the scipy stack (specifically
>> matplotlib and pandas and their dependencies) is really useful. But
>> conda is *not* what I'd use for day to day work, so being able to
>> install via pip is important to me.
>>
>
What if "conda install" did work for virtually all pypi packages? (one way
or the other) -- would you use and recommend Anaconda (or miniconda) then?


> It should be noted that installing
>> via pip *is* possible - via some of the relevant projects having
>> published wheels, and the rest being available via Christoph Gohlke's
>> site either as wheels or as wininsts that I can convert. But that's
>> not a seamless process, so it's not something I'd be too happy
>> explaining to a colleague should I want to share the workload for that
>> type of thing.
>>
>
right -- that could be made better right now -- or soon. Gohlke's packages
can't be simply put up on PyPi for licensing reasons (he's using the Intel
math libs). But some folks are working really hard on getting a numpy wheel
that will work virtually everywhere, and still give good performance for
numerics.  From there, the core SciPy  stack should follow (it's already on
PyPi for OS-X).

Which is a GREAT move in the right direction, but doesn't get us quite to
where PyPi can support the more complex packages.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150518/45be4bd0/attachment.html>