[Pandas-dev] DyND and pandas [was Rewriting some of internals of pandas in C/C++? / Roadmap]

[Wes McKinney]

On Tue, Jan 12, 2016 at 3:41 PM, Jeff Reback <jeffreback at gmail.com> wrote:
> So this thread is off-topic, but I believe the gist of what wes is proposing
> from a technical point of view for libpandas:
>
> - the user facing pandas API will not change (except better perf /
> copy-on-write etc)
> - the back-end API should not change much either
> - c-API for the back-end.
> - allows swappable / agnostic numpy-like back-ends.
> - ideally libpandas won't rewrite a completely new dtype system, maybe could
> co-op datashape / pluribus for extensible dtypes
>
> If the above are met by a back-end, e.g. numpy, potentially DyND, then it a
> back-end should be allowed
> (certainly as an optional dep, whether its required or not can be a choice
> made down the road).
>
> I think during implementation, that wes will be congnizant of these points,
> and leave things as wide open as
> possible w/o going down the road we are currently in (where lots of
> different API's are intermixed).
>

Yep, you nailed it.

> Jeff
>
>
> On Tue, Jan 12, 2016 at 6:20 PM, Irwin Zaid <izaid at continuum.io> wrote:
>>
>>
>>> This discussion doesn't belong on this mailing list, but a couple of
>>> brief points.
>>
>>
>> Wes, if you don't want this discussion on this mailing list then don't say
>> things like: "it still feels like a political quagmirie leftover from the
>> Continuum-Enthought rift in 2011". My email reply to that was simply a
>> statement of facts, as this one will also be.
>>
>>> I was approached by Travis and Peter about being a part of Continuum
>>> Analytics in late 2011. According to my e-mail records we were having
>>> these discussions at least as early as October 2011. The phrase "NumPy
>>> 2.0" was spoken in this epoch (referring to
>>> -the-project-now-known-as-DyND). So, I have quite a bit of first- and
>>> second-hand information from this time period, including many of the
>>> details of Mark's Enthought-sponsored NumPy development and the
>>> problems that occurred online and offline.
>>
>>
>> The phrase "NumPy 2.0" means a number of things, and DyND was not one of
>> them. Yes, you have some first-hand knowledge,
>> but it's not relevant. Even IF it was, a lot of modern DyND also came from
>> my massive contribution before I joined Continuum.
>>
>> Mark will speak up here as well.
>>
>>>
>>> I applaud Continuum for using R&D budget to build something new and
>>> forward thinking that is also permissively licensed open source
>>> software. However, it is well known that open source projects driven
>>> by for-profit organizations can run into governance problems that
>>> place them in conflict with the community. Since DyND is a large
>>> project that I would not be comfortable forking (if that were required
>>> in the future), building an outside developer and user community is
>>> essential if pandas is to consider using it as a hard dependency in
>>> the future.
>>>
>>> The Apache Software Foundation exists for this reason and others, and
>>> if you wish to place a community-oriented and merit-based governance
>>> structure around DyND to assist with its incubation, the ASF may be
>>> worth pursuing. NumFOCUS provides a fiscal sponsorship apparatus but
>>> does not really address the governance questions. Whether or not the
>>> governance issues are real doesn't really matter; it's about setting
>>> people's minds at ease.
>>
>>
>> Okay, let me state again: The majority of DyND's contributions (as net
>> from Mark, myself, and Ian) came without Continuum funding. Just because
>> Continuum is funding DyND now does not make it a "Continuum project",
>> whatever this means.
>>
>> Some of your other points are valid, and we'll address them as best we can
>> as time goes on. DyND clearly needs a community, but it's a chicken-and-egg
>> problem. If you try and build something hard, it takes time and users come
>> when things work.
>>
>> The issue of refactoring Pandas is a different one that I'll add comments
>> to in another email.
>>
>> Irwin
>>
>> _______________________________________________
>> Pandas-dev mailing list
>> Pandas-dev at python.org
>> https://mail.python.org/mailman/listinfo/pandas-dev
>>
>