[Numpy-discussion] A little about XND

[Travis Oliphant]

Hi everyone,

I'm glad I'm able to contribute back to this discussion thread.  I wanted
to post a quick message to this group to make sure there is no
mis-information about XND which has finally reached the point where it can
be experimented with (http://xnd.io) and commented on.

XND came out of thoughts and conversations we had at Continuum (now
Anaconda) when thinking about cross-language array computing and how to
enable improved features for high-level users in many languages (including
Python, R, Ruby, Node, Scala, Rust, Go, etc.).  Technically there are three
projects that make up XND (thus the name Plures for the Github
organization). All of these projects have a C-library and then a high-level
interface (right now we only have resources to develop the Python interface
but would love to see support for other languages).  xnd (libxnd) is the
typed container.  ndtypes (libndtypes) is the (datashape-like) type system
with a grammar, parser, and type matcher.  gumath (libgumath) are
generalized ufuncs which represent the entire function system on xnd.

We will be talking more about XND in the coming months and years, but for
the purposes of this list, I wanted to make it clear that

1) XND is not trying to replace NumPy. XND is a low-level library and
intended to be such. It would be most welcome if someday NumPy uses XND.
We understand this may be a while and certainly not before NumPy 2.0 or
3.0.

2) Our initial target users are Numba, pandas, Dask, xarray, and other
higher-level objects at the moment. We are eagerly searching for
integration opportunities to connect more developers (or advanced users) to
xnd before making more progress.

3) We do discuss array-like things in the public channels. NumPy users and
developers are welcome in those channels.  Everything is done in public
including the weekly meeting which anyone can attend:

Weekly meeting:  meet.google.com/heo-fmow-omz

Live discussions:  https://gitter.im/Plures/xnd for the libraries
themselves
                             https://gitter.im/Plures/xnd-ml for
integrations.

Issues and PRs: https://github.com/plures  --- under the various projects.

4) We are thinking about adding a custom-dtype to NumPy that uses xnd and
would be happy for anyone's help on that project.

5) We are in the early stages of exploring a high-level array interface
(using the ideas of MoA and the Psi Calculus with Lenore Mullen who worked
on APL years ago).  Likely the first place this will see some initial
progress is in an ND Sparse array that uses XND.

We welcome participation and input from all.  Stefan Krah has written the
majority of the code and so we tend to respect his point of view.  Pearu
Peterson (of f2py and SciPy fame) has made some useful contributions
recently.

Stefan and I have been talking roughly weekly for a couple of years and so
some of the problems currently there, I am certainly responsible for.

Two of our immediate goals are to work with the Numba team to get support
for ndtypes in Numba and allow Numba to use libgumath in no-python mode.

I look forward to continuing the conversation with any of you who want to
participate.  Perhaps some of us can meet up during NumPy sprints to
discuss more.

XND is also currently looking for funding and time from interested parties
to continue its development.

-Travis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20180617/44d1f72b/attachment.html>