[Pandas-dev] DyND and pandas [was Rewriting some of internals of pandas in C/C++? / Roadmap]

[Jeff Reback]

So this thread is off-topic, but I believe the gist of what wes is
proposing from a technical point of view for libpandas:

- the user facing pandas API will not change (except better perf /
copy-on-write etc)
- the back-end API should not change much either
- c-API for the back-end.
- allows swappable / agnostic numpy-like back-ends.
- ideally libpandas won't rewrite a completely new dtype system, maybe
could co-op datashape / pluribus for extensible dtypes

If the above are met by a back-end, e.g. numpy, potentially DyND, then it a
back-end should be allowed
(certainly as an optional dep, whether its required or not can be a choice
made down the road).

I think during implementation, that wes will be congnizant of these points,
and leave things as wide open as
possible w/o going down the road we are currently in (where lots of
different API's are intermixed).

Jeff


On Tue, Jan 12, 2016 at 6:20 PM, Irwin Zaid <izaid at continuum.io> wrote:

>
> This discussion doesn't belong on this mailing list, but a couple of
>> brief points.
>>
>
> Wes, if you don't want this discussion on this mailing list then don't say
> things like: "it still feels like a political quagmirie leftover from the
> Continuum-Enthought rift in 2011". My email reply to that was simply a
> statement of facts, as this one will also be.
>
> I was approached by Travis and Peter about being a part of Continuum
>> Analytics in late 2011. According to my e-mail records we were having
>> these discussions at least as early as October 2011. The phrase "NumPy
>> 2.0" was spoken in this epoch (referring to
>> -the-project-now-known-as-DyND). So, I have quite a bit of first- and
>> second-hand information from this time period, including many of the
>> details of Mark's Enthought-sponsored NumPy development and the
>> problems that occurred online and offline.
>>
>
> The phrase "NumPy 2.0" means a number of things, and DyND was not one of
> them. Yes, you have some first-hand knowledge,
> but it's not relevant. Even IF it was, a lot of modern DyND also came from
> my massive contribution before I joined Continuum.
>
> Mark will speak up here as well.
>
>
>> I applaud Continuum for using R&D budget to build something new and
>> forward thinking that is also permissively licensed open source
>> software. However, it is well known that open source projects driven
>> by for-profit organizations can run into governance problems that
>> place them in conflict with the community. Since DyND is a large
>> project that I would not be comfortable forking (if that were required
>> in the future), building an outside developer and user community is
>> essential if pandas is to consider using it as a hard dependency in
>> the future.
>>
>> The Apache Software Foundation exists for this reason and others, and
>> if you wish to place a community-oriented and merit-based governance
>> structure around DyND to assist with its incubation, the ASF may be
>> worth pursuing. NumFOCUS provides a fiscal sponsorship apparatus but
>> does not really address the governance questions. Whether or not the
>> governance issues are real doesn't really matter; it's about setting
>> people's minds at ease.
>>
>
> Okay, let me state again: The majority of DyND's contributions (as net
> from Mark, myself, and Ian) came without Continuum funding. Just because
> Continuum is funding DyND now does not make it a "Continuum project",
> whatever this means.
>
> Some of your other points are valid, and we'll address them as best we can
> as time goes on. DyND clearly needs a community, but it's a chicken-and-egg
> problem. If you try and build something hard, it takes time and users come
> when things work.
>
> The issue of refactoring Pandas is a different one that I'll add comments
> to in another email.
>
> Irwin
>
> _______________________________________________
> Pandas-dev mailing list
> Pandas-dev at python.org
> https://mail.python.org/mailman/listinfo/pandas-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pandas-dev/attachments/20160112/2231ce3f/attachment.html>