From cwg at falma.de  Mon Oct  1 13:43:00 2012
From: cwg at falma.de (Christoph Groth)
Date: Mon, 01 Oct 2012 13:43:00 +0200
Subject: [Cython] import relative cimport
Message-ID: <87ipaumn97.fsf@falma.de>

Hello,

I believe that relative cimports would be useful in Cython.  (For a
motivation, see my recent posting on cython-user
http://thread.gmane.org/gmane.comp.python.cython.user/7401)

So I tried to assess how much work it would be to implement that
feature, but that's a tough job as I do not have a good overview of
Cython's considerable code base.  Would allowing relative cimports be a
rather small thing comparable to commit 599c34053 which introduced
relative imports
(http://codereview.appspot.com/1734044/patch/2001/3001)?  Or are there
any issues that require serious refactoring?

Christoph


From cwg at falma.de  Mon Oct  1 13:56:40 2012
From: cwg at falma.de (Christoph Groth)
Date: Mon, 01 Oct 2012 13:56:40 +0200
Subject: [Cython] import relative cimport
References: <87ipaumn97.fsf@falma.de>
Message-ID: <87ehlimmmf.fsf@falma.de>

Oh, something I forgot:

in Cython/Compiler/Parsing.py I find code that emits "Relative cimport
is not supported yet".  However, when I try to compile code which
includes a relative cimport, I do not get this error, but instead

from .graph.defs cimport gint
^
------------------------------------------------------------

kwant/_system.pyx:8:0: 'graph.defs.pxd' not found

Is this related to the behavior of Pyrex?  I tried creating a file
"graph.defs.pxd" but I was unable to make the above line compile.


From greg.ewing at canterbury.ac.nz  Tue Oct  2 00:12:07 2012
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 02 Oct 2012 11:12:07 +1300
Subject: [Cython] import relative cimport
In-Reply-To: <87ehlimmmf.fsf@falma.de>
References: <87ipaumn97.fsf@falma.de> <87ehlimmmf.fsf@falma.de>
Message-ID: <506A1537.8090909@canterbury.ac.nz>

Christoph Groth wrote:

> in Cython/Compiler/Parsing.py I find code that emits "Relative cimport
> is not supported yet".

> Is this related to the behavior of Pyrex?

There's no such code in Pyrex's Parsing.py, and if I try to
compile this with Pyrex:

    from .bar.stuff cimport thing

I get

    /Users/greg/foo/foo.pyx:1:5: Expected an identifier

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Oct  2 00:14:30 2012
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 02 Oct 2012 11:14:30 +1300
Subject: [Cython] import relative cimport
In-Reply-To: <87ehlimmmf.fsf@falma.de>
References: <87ipaumn97.fsf@falma.de> <87ehlimmmf.fsf@falma.de>
Message-ID: <506A15C6.2030403@canterbury.ac.nz>

Christoph Groth wrote:

> kwant/_system.pyx:8:0: 'graph.defs.pxd' not found

That part *is* related to Pyrex -- it's a holdover from
the old method of dealing with .pyx files in packages.
It doesn't have anything to do with relative imports,
though.

-- 
Greg

From robertwb at gmail.com  Tue Oct  2 02:08:45 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 1 Oct 2012 17:08:45 -0700
Subject: [Cython] import relative cimport
In-Reply-To: <87ipaumn97.fsf@falma.de>
References: <87ipaumn97.fsf@falma.de>
Message-ID: <CADiQ+QDQQQyDTg_XD0REW5-g9OjRrbiP6oDAYSWDZyhf7sDK+w@mail.gmail.com>

On Mon, Oct 1, 2012 at 4:43 AM, Christoph Groth <cwg at falma.de> wrote:
> Hello,
>
> I believe that relative cimports would be useful in Cython.  (For a
> motivation, see my recent posting on cython-user
> http://thread.gmane.org/gmane.comp.python.cython.user/7401)
>
> So I tried to assess how much work it would be to implement that
> feature, but that's a tough job as I do not have a good overview of
> Cython's considerable code base.  Would allowing relative cimports be a
> rather small thing comparable to commit 599c34053 which introduced
> relative imports
> (http://codereview.appspot.com/1734044/patch/2001/3001)?  Or are there
> any issues that require serious refactoring?

This might be a simple as letting the import code (e.g. at
https://github.com/cython/cython/blob/master/Cython/Utility/ImportExport.c
and everywhere it's called) allow and use relative imports. At a high
level, the tricky part is that cimports are resolved at Cython
complile time, then again at runtime. IIRC there is also some name
unification that uses the absolute path (which might not turn out to
matter, but something to be aware of and look into).

In short, I don't think it'd be that hard, but it'd probably touch a
larger spread of the codebase than the relative (non-c)import case.

- Robert

From dalcinl at gmail.com  Tue Oct  2 17:06:22 2012
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Oct 2012 12:06:22 -0300
Subject: [Cython] porting mpi4py to PyPy runtime
Message-ID: <CAEcYPwBC5Pdap1XaVrW+H=jdPubyLvvY8hQBPdG+kuzCG4BwQQ@mail.gmail.com>

After successfully porting mpi4py to PyPy 1.9, I want to share a few
issues I've encountered. The most serious onesare on PyPy's side, but
Cython do require a cuple of very minor fixes.

1) Cython uses _PyLong_AsByteArray() to convert integers of unknown
sizeof larger than long long. I had to cinclude the following code to
get the C sources compile:

static int
_PyLong_AsByteArray(PyLongObject* v, unsigned char* bytes, size_t n,
int little_endian, int is_signed)
{
  PyErr_SetString(PyExc_RuntimeError,
                  "PyPy: _PyLong_AsByteArray() not available");
  return -1;
}


2) PyPy's implementation of PyBuffer_FillInfo() has a bug, it ignores
the "readonly" parameter instead of generating an error if readonly=1
and PyBUF_WRITABLE is on. I had to to cinclude the following code to
monkeypatch it:

static int
PyBuffer_FillInfo_PyPy(Py_buffer *view, PyObject *obj, void *buf,
Py_ssize_t len, int readonly, int flags)
{
  if (view == NULL) return 0;
  if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) &&
      (readonly == 1)) {
    PyErr_SetString(PyExc_BufferError,
                    "Object is not writable.");
    return -1;
  }
  if (PyBuffer_FillInfo(view, obj, buf, len, readonly, flags) < 0)
    return -1;
  view->readonly = readonly;
  return 0;
}
#define PyBuffer_FillInfo PyBuffer_FillInfo_PyPy


3) In PyPy, PyCode_GetNumFree() accepts a PyObject*, not
PyCodeObject*. I've monkeypatched as follow:

#define PyCode_GetNumFree(o) PyCode_GetNumFree((PyObject *)(o))


4) mpi4py uses @classmethod decorator for many methods in cdef
classes. I really do not know what's going on, but method lookups do
not return the classmethod, but the original "unwrapped" method. I've
found a way to trick it, but again, I have no idea why it is working.
Basically, I have to execute hasattr() (in Python code, not Cython
code) on every class method: The hack reduces to adding the following
code at the very end of Cython code (note that I'm using "exec"):

if PYPY: exec """
def _pypy_setup():
    for klass in (
        Status,
        Datatype,
        Request,
        Message,
        Op,
        Group,
        Info,
        Errhandler,
        Comm,
        Win,
        File,
        ):
        for name in klass.__dict__:
            meth = klass.__dict__[name]
            if (isinstance(meth, classmethod) or
                isinstance(meth, staticmethod)):
                hasattr(klass, name)
_pypy_setup()
del _pypy_setup
"""


I think (1) and (3) can be trivially handled in Cython, (3) is not an
issue for Cython as PyBuffer_FillInfo() is not directly used. And
about (4), we really need help from PyPy folks, they implement
PyType_Modified() as a non-op
(https://bitbucket.org/pypy/pypy/src/release-1.9/pypy/module/cpyext/typeobject.py#cl-726)
but that is not playing well with Cython.

Below, the relevant pieces of code in mpi4py containing all the code
I've described previously.

https://code.google.com/p/mpi4py/source/browse/src/atimport.h#324
https://code.google.com/p/mpi4py/source/browse/src/MPI/MPI.pyx#280

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Tue Oct  2 21:33:09 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Oct 2012 21:33:09 +0200
Subject: [Cython] porting mpi4py to PyPy runtime
In-Reply-To: <CAEcYPwBC5Pdap1XaVrW+H=jdPubyLvvY8hQBPdG+kuzCG4BwQQ@mail.gmail.com>
References: <CAEcYPwBC5Pdap1XaVrW+H=jdPubyLvvY8hQBPdG+kuzCG4BwQQ@mail.gmail.com>
Message-ID: <506B4175.2070700@behnel.de>

Am 02.10.12 17:06, schrieb Lisandro Dalcin:
> After successfully porting mpi4py to PyPy 1.9

Very cool, and thanks for reporting on it.


>, I want to share a few
> issues I've encountered. The most serious ones are on PyPy's side

Definitely.


>, but Cython do require a cuple of very minor fixes.
> 
> 1) Cython uses _PyLong_AsByteArray() to convert integers of unknown
> sizeof larger than long long. I had to cinclude the following code to
> get the C sources compile:
> 
> static int
> _PyLong_AsByteArray(PyLongObject* v, unsigned char* bytes, size_t n,
> int little_endian, int is_signed)
> {
>   PyErr_SetString(PyExc_RuntimeError,
>                   "PyPy: _PyLong_AsByteArray() not available");
>   return -1;
> }

Looks like we should just special case the conversion for PyPy here so
that it stops using that function.


> 2) PyPy's implementation of PyBuffer_FillInfo() has a bug, it ignores
> the "readonly" parameter instead of generating an error if readonly=1
> and PyBUF_WRITABLE is on. I had to to cinclude the following code to
> monkeypatch it:
> 
> static int
> PyBuffer_FillInfo_PyPy(Py_buffer *view, PyObject *obj, void *buf,
> Py_ssize_t len, int readonly, int flags)
> {
>   if (view == NULL) return 0;
>   if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) &&
>       (readonly == 1)) {
>     PyErr_SetString(PyExc_BufferError,
>                     "Object is not writable.");
>     return -1;
>   }
>   if (PyBuffer_FillInfo(view, obj, buf, len, readonly, flags) < 0)
>     return -1;
>   view->readonly = readonly;
>   return 0;
> }
> #define PyBuffer_FillInfo PyBuffer_FillInfo_PyPy

I would prefer it if they fixed that one on their side. I assume you
filed a bug report on their tracker?


> 3) In PyPy, PyCode_GetNumFree() accepts a PyObject*, not
> PyCodeObject*. I've monkeypatched as follow:
> 
> #define PyCode_GetNumFree(o) PyCode_GetNumFree((PyObject *)(o))

Cython could do that as well, although I think PyPy should fix it
themselves. AFAIK, the current PyPy C-API isn't versioned (at least not
independently of the PyPy version), so applying such a work-around on
our side would mean that it's going to fail again at some point when
they fix it.

Still, if it makes things work for the time being, so be it...


> 4) mpi4py uses @classmethod decorator for many methods in cdef
> classes. I really do not know what's going on, but method lookups do
> not return the classmethod, but the original "unwrapped" method. I've
> found a way to trick it, but again, I have no idea why it is working.
> Basically, I have to execute hasattr() (in Python code, not Cython
> code) on every class method: The hack reduces to adding the following
> code at the very end of Cython code (note that I'm using "exec"):
> 
> if PYPY: exec """
> def _pypy_setup():
>     for klass in (
>         Status,
>         Datatype,
>         Request,
>         Message,
>         Op,
>         Group,
>         Info,
>         Errhandler,
>         Comm,
>         Win,
>         File,
>         ):
>         for name in klass.__dict__:
>             meth = klass.__dict__[name]
>             if (isinstance(meth, classmethod) or
>                 isinstance(meth, staticmethod)):
>                 hasattr(klass, name)
> _pypy_setup()
> del _pypy_setup
> """
> 
> 
> I think (1) and (3) can be trivially handled in Cython, (3) is not an
> issue for Cython as PyBuffer_FillInfo() is not directly used. And
> about (4), we really need help from PyPy folks, they implement
> PyType_Modified() as a non-op
> (https://bitbucket.org/pypy/pypy/src/release-1.9/pypy/module/cpyext/typeobject.py#cl-726)
> but that is not playing well with Cython.

PyType_Ready() is also incomplete.


> Below, the relevant pieces of code in mpi4py containing all the code
> I've described previously.
> 
> https://code.google.com/p/mpi4py/source/browse/src/atimport.h#324
> https://code.google.com/p/mpi4py/source/browse/src/MPI/MPI.pyx#280

It would be nice if you could write up pull requests for the necessary
work-arounds above.

Stefan


From whosaysni at gmail.com  Fri Oct  5 09:01:42 2012
From: whosaysni at gmail.com (Yasushi Masuda)
Date: Fri, 5 Oct 2012 16:01:42 +0900
Subject: [Cython] Translating docs into Japanese
Message-ID: <CAO0wvAtOKihLz5Jdqjm-3Q_L6FKvXhJc0ZFmqVMiE2Qr2j=xYA@mail.gmail.com>

Hello list,

Recently I've worked on translating cython documents into Japanese,
and just finished for quickstart/tutorial.
It was mainly for me and my colleagues, but I think it would be useful
for all cython users speaking Japanese, so I'd like to publish it via
my company's unofficial site:

http://omake.accense.com/static/doc-ja/cython/

I suppose that docs are permissible for translation, redistribution
and publish, but if you authors of original docs have troubles, please
let me know...

Thanks for great library (and, of course, documents)!

-- 
Yasushi Masuda
http://ymasuda.jp/
whosaysni at twitter/gmail

From stefan_ml at behnel.de  Sat Oct  6 15:56:56 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 06 Oct 2012 15:56:56 +0200
Subject: [Cython] Translating docs into Japanese
In-Reply-To: <CAO0wvAtOKihLz5Jdqjm-3Q_L6FKvXhJc0ZFmqVMiE2Qr2j=xYA@mail.gmail.com>
References: <CAO0wvAtOKihLz5Jdqjm-3Q_L6FKvXhJc0ZFmqVMiE2Qr2j=xYA@mail.gmail.com>
Message-ID: <507038A8.9040209@behnel.de>

[moving this to cython-users]

Yasushi Masuda, 05.10.2012 09:01:
> Recently I've worked on translating cython documents into Japanese,
> and just finished for quickstart/tutorial.
> It was mainly for me and my colleagues, but I think it would be useful
> for all cython users speaking Japanese, so I'd like to publish it via
> my company's unofficial site:
> 
> http://omake.accense.com/static/doc-ja/cython/
> 
> I suppose that docs are permissible for translation, redistribution
> and publish, but if you authors of original docs have troubles, please
> let me know...
> 
> Thanks for great library (and, of course, documents)!

Thanks for the translation.

I don't see any problems with you providing it, especially on your own web
site. However, could you please add a link back to the original
documentation? Something like "This is a translation of the original Cython
documentation from [doclink here]". Just in case your site gets outdated at
some point (happens...) and also to prevent readers from misinterpreting
your relation to the Cython project.

@others: should we link to this site from the Cython home page?

Stefan


From stefan_ml at behnel.de  Sun Oct  7 08:05:12 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Oct 2012 08:05:12 +0200
Subject: [Cython] [cython-users] beginner question: passing std::vector
	by reference
In-Reply-To: <CADiQ+QCH7j+uyG00nH3eCiQVmAfo9fhLQUoKzP5AO_ihB=Jraw@mail.gmail.com>
References: <e86b0674-dad9-4ff3-9f39-61829a9b9378@googlegroups.com>
	<CADiQ+QCH7j+uyG00nH3eCiQVmAfo9fhLQUoKzP5AO_ihB=Jraw@mail.gmail.com>
Message-ID: <50711B98.4000809@behnel.de>

Robert Bradshaw, 03.10.2012 06:55:
> On Tue, Oct 2, 2012 at 12:38 AM, peter aberline wrote:
>> Please excuse my basic question. I'm trying to pass a reference to a
>> std::vector to a function and I'm getting a cpp compilaton error. Here's a
>> minimum example of what I'm doing:
>>
>> test.py:
>>
>> import sys
>>
>> def vectortest(a, idx):
>>     print ('a[idx] is: ' + str(a[idx]))
>>     a[idx] += 1
>>     print ('a[idx] is: ' + str(a[idx]))
>>     return a
>>
>> def run():
>>     lst = range(1, 5)
>>     lstNew = vectortest(lst, 1)
>>
>>     print ('lstNew[1] is :' + str(lstNew[1]))
>>
>> test.pxd
>> import cython
>> from libcpp.vector cimport vector
>>
>> import sys
>>
>> if sys.version_info < (3,):
>>     range = xrange
>>
>> cpdef vector[int] vectortest(vector[int] a, int idx)  # <-- This works fine
>> ('a' is pass by value?)
>>
>> #cpdef vector[int] vectortest(vector[int]& a, int idx) #<-- This generates
>> an 'uninitialised reference' cpp compilation error ('a' is pass by
>> reference?')
>>
>> @cython.locals(lst = vector[int])
>> cpdef run()
>>
>>
>> Compilation error:
>>
>> $/test.cpp: In function ?PyObject* __pyx_pw_4perf_1vectortest(PyObject*,
>> PyObject*, PyObject*)?:
>> $/test.cpp:640:21: error: ?__pyx_v_a? declared as reference but not
>> initialized
> 
> Looks like this was a bug introduced In 0.17.1 and already fixed in master.

In that case, the fix (whatever it was) should also go into the 0.17 branch
so that we can release within a suitable time frame.

Stefan


From whosaysni at gmail.com  Sun Oct  7 12:32:42 2012
From: whosaysni at gmail.com (Yasushi Masuda)
Date: Sun, 7 Oct 2012 03:32:42 -0700 (PDT)
Subject: [Cython] Translating docs into Japanese
In-Reply-To: <507038A8.9040209@behnel.de>
References: <CAO0wvAtOKihLz5Jdqjm-3Q_L6FKvXhJc0ZFmqVMiE2Qr2j=xYA@mail.gmail.com>
	<507038A8.9040209@behnel.de>
Message-ID: <3fc50185-fb8d-48dd-955b-3b64cec08da9@googlegroups.com>

Stefan,

Thanks for reply - I added notice/link for original docs, both on toppage 
of the document and footer in every page.

Yasushi

2012?10?6???? 22?57?08? UTC+9 Stefan Behnel:
>
> [moving this to cython-users] 
>
> Yasushi Masuda, 05.10.2012 09:01: 
> > Recently I've worked on translating cython documents into Japanese, 
> > and just finished for quickstart/tutorial. 
> > It was mainly for me and my colleagues, but I think it would be useful 
> > for all cython users speaking Japanese, so I'd like to publish it via 
> > my company's unofficial site: 
> > 
> > http://omake.accense.com/static/doc-ja/cython/ 
> > 
> > I suppose that docs are permissible for translation, redistribution 
> > and publish, but if you authors of original docs have troubles, please 
> > let me know... 
> > 
> > Thanks for great library (and, of course, documents)! 
>
> Thanks for the translation. 
>
> I don't see any problems with you providing it, especially on your own web 
> site. However, could you please add a link back to the original 
> documentation? Something like "This is a translation of the original 
> Cython 
> documentation from [doclink here]". Just in case your site gets outdated 
> at 
> some point (happens...) and also to prevent readers from misinterpreting 
> your relation to the Cython project. 
>
> @others: should we link to this site from the Cython home page? 
>
> Stefan 
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20121007/42efc3ca/attachment.html>

From robertwb at gmail.com  Thu Oct 11 19:25:25 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 11 Oct 2012 10:25:25 -0700
Subject: [Cython] Fwd: [cython-users] can pointers be stored in Python's
 dict or list?
In-Reply-To: <5054149A.8040009@behnel.de>
References: <20120914161147.58f58ab8@dino>
	<CADiQ+QAeQC_orTB_YecH0_2ZEh68aCrR8MBpWLxfpLoFFn+L1g@mail.gmail.com>
	<20120914205458.09da1b0e@dino>
	<CADiQ+QCue=OYqVxsPm0tHhwuNA=ER_LxaAPM4j3uPV05Jbp7vg@mail.gmail.com>
	<CADiQ+QApwSt6T_TutnDMeOhrTSfhbexw8=+AtyDoYM5MDa_fSw@mail.gmail.com>
	<aa75f415-7197-442a-977e-1851891e2af4@email.android.com>
	<CADiQ+QBw7eTAZvp6qm5PVcT4-EDOd9RGiR=e3Tmt8E8GMWx5Dg@mail.gmail.com>
	<5054149A.8040009@behnel.de>
Message-ID: <CADiQ+QAqUeTStZpvryTTUiRiW00qCdc1J8nijUG4ynxfP8KOmg@mail.gmail.com>

Don't know how I missed this earlier.

On Fri, Sep 14, 2012 at 10:39 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 15.09.2012 00:39:
>> On Fri, Sep 14, 2012 at 2:29 PM, Dag Sverre Seljebotn wrote:
>>> Isn't there a case for converting to/from ctypes pointers rather than
>>> capsules? And if capsules, what would the secret word be? Hmm...
>>
>> +1, ctypes would be even better. It's not in the standard library
>> 'till 2.5, but I don't think that's a huge blocker as it can be
>> installed earlier.
>
> I'm not entirely sure I see the use case for starting to support the ctypes
> type system. Is there more to it than just passing pointers through Python
> code? A capsule (or even a special Cython extension type) sounds like a
> better option. For example, it's rather unlikely that non-Cython user code
> will start to support passing in ctypes types, so it would rather be a
> Cython-to-Cython-only thing anyway.

My assumption is that it would allow more than just Cython-to-Cython
translation. Essentially ctypes would be the lingua franca of, well,
external C types like pointers and any library that needs (say) and
int*, whether in Cython or not, could communicate this via Python
space.

I'ts just an idea, I haven't pursued it that far. The other option is
forgoing type safety or rolling our own (which might be lightweight
enough to just do).

> Do you assume that users would want to access C internals through the
> values that Cython returns to them? That sounds risky.

True, but it could be useful as well.

>>> Robert Bradshaw wrote:
>>>> Given the CObject vs Capsule differences in Py2 vs Py3, and the
>>>> clumsiness in using them (see below), any thoughts on automatically
>>>> converting void* to/from a Python object? There is the sticky issue of
>>>> pointer lifetime management, but we could assume the lifetime is
>>>> managed entirely in C in most cases.
>>>>
>>>> Note that <void*>object already has a specific meaning that we can't
>>>> hijack, but perhaps non-void* would be fine to do implicitly? Also, if
>>>> we built this into the language, could we provide type safety via the
>>>> name/description attribute? Something like this could be really handy
>>>> to have and isn't as easily done as a library (especially given the
>>>> 2/3 differences).
>
> We could support both use cases by adding both types to the type system.
> For example:
>
>     from cython import capsule
>
>     cap = <capsule[description]>some_ptr
>
> or, likely nicer:
>
>     cap = capsule(some_ptr, description)
>
> with auto-fallback to CObject on older Py2 versions. I'm not sure about the
> way back. Maybe a function like "decapsule(cap, description)" would work here?
>
> As for ctypes, we might get away with providing a generic <ctypes> cast and
> do the type matching ourselves for the forward way. For the way back into
> Cython types, however, we should require an explicitly typed ctypes
> variable. Just allowing a cast from anything to a Cython C type would let
> our type conversion code explode.
>
> Not sure how much work it would be to implement this, though. The way from
> Cython to ctypes may just be a simple mapping of type names, whereas the
> other way might require some utility code overhead.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Fri Oct 12 09:36:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Oct 2012 09:36:46 +0200
Subject: [Cython] array expressions
In-Reply-To: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
Message-ID: <5077C88E.1030205@behnel.de>

mark florisson, 24.08.2012 20:40:
> Here a pull request for element-wise array expressions for Cython:
> https://github.com/cython/cython/pull/144

Mark, any news on this? I'd like to see a version merged before the master
branch starts diverging all too far - it already requires a bit of adaptation.

Did you manage to split off a separate minivect package?

Stefan


From markflorisson88 at gmail.com  Fri Oct 12 12:14:42 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Fri, 12 Oct 2012 11:14:42 +0100
Subject: [Cython] array expressions
In-Reply-To: <5077C88E.1030205@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
Message-ID: <CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>

On 12 October 2012 08:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 24.08.2012 20:40:
>> Here a pull request for element-wise array expressions for Cython:
>> https://github.com/cython/cython/pull/144
>
> Mark, any news on this? I'd like to see a version merged before the master
> branch starts diverging all too far - it already requires a bit of adaptation.
>
> Did you manage to split off a separate minivect package?
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

I have some time this weekend, I'll see if I can finish it then. Shall
I make a new PR or merge it directly?

From stefan_ml at behnel.de  Fri Oct 12 12:28:46 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Oct 2012 12:28:46 +0200
Subject: [Cython] array expressions
In-Reply-To: <CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
Message-ID: <5077F0DE.6090106@behnel.de>

mark florisson, 12.10.2012 12:14:
> On 12 October 2012 08:36, Stefan Behnel wrote:
>> mark florisson, 24.08.2012 20:40:
>>> Here a pull request for element-wise array expressions for Cython:
>>> https://github.com/cython/cython/pull/144
>>
>> Mark, any news on this? I'd like to see a version merged before the master
>> branch starts diverging all too far - it already requires a bit of adaptation.
>>
>> Did you manage to split off a separate minivect package?
> 
> I have some time this weekend, I'll see if I can finish it then. Shall
> I make a new PR or merge it directly?

As I said, I won't review it anyway, so it's no difference for me. But
given that your original pull request didn't receive any comments, I doubt
that making a new one would serve a major purpose.

Stefan


From robertwb at gmail.com  Fri Oct 12 17:50:54 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Fri, 12 Oct 2012 08:50:54 -0700
Subject: [Cython] array expressions
In-Reply-To: <CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
Message-ID: <CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>

On Fri, Oct 12, 2012 at 3:14 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 12 October 2012 08:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 24.08.2012 20:40:
>>> Here a pull request for element-wise array expressions for Cython:
>>> https://github.com/cython/cython/pull/144
>>
>> Mark, any news on this? I'd like to see a version merged before the master
>> branch starts diverging all too far - it already requires a bit of adaptation.
>>
>> Did you manage to split off a separate minivect package?

I'm assuming this has already been looked at, at least to some level,
by Dag, but I'll try to take a brief pass at it too (probably more the
interface than the implementation).

I don't see a reason for a new pull request.

- Robert

From d.s.seljebotn at astro.uio.no  Fri Oct 12 21:01:43 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Oct 2012 21:01:43 +0200
Subject: [Cython] array expressions
In-Reply-To: <CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
Message-ID: <50786917.8000303@astro.uio.no>

On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 12 October 2012 08:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 24.08.2012 20:40:
>>>> Here a pull request for element-wise array expressions for Cython:
>>>> https://github.com/cython/cython/pull/144
>>>
>>> Mark, any news on this? I'd like to see a version merged before the master
>>> branch starts diverging all too far - it already requires a bit of adaptation.
>>>
>>> Did you manage to split off a separate minivect package?
>
> I'm assuming this has already been looked at, at least to some level,
> by Dag, but I'll try to take a brief pass at it too (probably more the
> interface than the implementation).

Thanks for doing that, it'd be great to get this in (but myself I've got 
nothing to spare). I'll admit I was mostly focused on the generated code 
and the algorithms in minivect rather than the integration with Cython.

Dag Sverre


>
> I don't see a reason for a new pull request.
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>


From markflorisson88 at gmail.com  Sat Oct 13 20:30:50 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sat, 13 Oct 2012 19:30:50 +0100
Subject: [Cython] array expressions
In-Reply-To: <50786917.8000303@astro.uio.no>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
Message-ID: <CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>

On 12 October 2012 20:01, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>
>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>>
>>> On 12 October 2012 08:36, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>
>>>> mark florisson, 24.08.2012 20:40:
>>>>>
>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>> https://github.com/cython/cython/pull/144
>>>>
>>>>
>>>> Mark, any news on this? I'd like to see a version merged before the
>>>> master
>>>> branch starts diverging all too far - it already requires a bit of
>>>> adaptation.
>>>>
>>>> Did you manage to split off a separate minivect package?
>>
>>
>> I'm assuming this has already been looked at, at least to some level,
>> by Dag, but I'll try to take a brief pass at it too (probably more the
>> interface than the implementation).
>
>
> Thanks for doing that, it'd be great to get this in (but myself I've got
> nothing to spare). I'll admit I was mostly focused on the generated code and
> the algorithms in minivect rather than the integration with Cython.
>
> Dag Sverre
>
>
>
>>
>> I don't see a reason for a new pull request.
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Great. As for the packaging, I'm creating a distribution branch, and a
subtree branch. Newer versions of git have a 'subtree' command
(previously https://github.com/apenwarr/git-subtree), which allows one
to split of, merge, push, and pull subdirectories.

This means when users pull the master project, they get the
sub-projects as well (without themselves needing newer git versions).
Any changes to a subproject can be merged into the subproject, and
changes can be pulled back in (with a squash option to avoid mixing in
the subproject's history).

What about using this approach? That way Cython remains stable and
pinned on the right minivect version now and in the future, with no
burden on users.

From stefan_ml at behnel.de  Sun Oct 14 08:18:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 08:18:27 +0200
Subject: [Cython] array expressions
In-Reply-To: <CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
Message-ID: <507A5933.90309@behnel.de>

mark florisson, 13.10.2012 20:30:
> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>> mark florisson, 24.08.2012 20:40:
>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>> https://github.com/cython/cython/pull/144
>>>>>
>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>> the master branch starts diverging all too far - it already
>>>>> requires a bit of adaptation.
>>>>> Did you manage to split off a separate minivect package?
>>>
>>> I'm assuming this has already been looked at, at least to some level,
>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>> interface than the implementation).
>>
>> Thanks for doing that, it'd be great to get this in (but myself I've got
>> nothing to spare). I'll admit I was mostly focused on the generated code and
>> the algorithms in minivect rather than the integration with Cython.
>>
>>> I don't see a reason for a new pull request.
> 
> Great. As for the packaging, I'm creating a distribution branch, and a
> subtree branch. Newer versions of git have a 'subtree' command
> (previously https://github.com/apenwarr/git-subtree), which allows one
> to split of, merge, push, and pull subdirectories.
> 
> This means when users pull the master project, they get the
> sub-projects as well (without themselves needing newer git versions).
> Any changes to a subproject can be merged into the subproject, and
> changes can be pulled back in (with a squash option to avoid mixing in
> the subproject's history).
> 
> What about using this approach? That way Cython remains stable and
> pinned on the right minivect version now and in the future, with no
> burden on users.

I still prefer having separate packages. I mean, we don't ship NumPy
either, even though a lot of people use Cython together with it.

Keeping the two packages separate helps in keeping the interface between
both clean. I wouldn't want to end up with Cython shipping some patched up
version of minivect just because it's so easy, and I would like to allow
users to install a new version of either Cython or minivect at any time.

Stefan


From d.s.seljebotn at astro.uio.no  Sun Oct 14 10:18:36 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sun, 14 Oct 2012 10:18:36 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A5933.90309@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de>
Message-ID: <507A755C.30802@astro.uio.no>

On 10/14/2012 08:18 AM, Stefan Behnel wrote:
> mark florisson, 13.10.2012 20:30:
>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>
>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>> the master branch starts diverging all too far - it already
>>>>>> requires a bit of adaptation.
>>>>>> Did you manage to split off a separate minivect package?
>>>>
>>>> I'm assuming this has already been looked at, at least to some level,
>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>> interface than the implementation).
>>>
>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>> nothing to spare). I'll admit I was mostly focused on the generated code and
>>> the algorithms in minivect rather than the integration with Cython.
>>>
>>>> I don't see a reason for a new pull request.
>>
>> Great. As for the packaging, I'm creating a distribution branch, and a
>> subtree branch. Newer versions of git have a 'subtree' command
>> (previously https://github.com/apenwarr/git-subtree), which allows one
>> to split of, merge, push, and pull subdirectories.
>>
>> This means when users pull the master project, they get the
>> sub-projects as well (without themselves needing newer git versions).
>> Any changes to a subproject can be merged into the subproject, ands.
>> changes can be pulled back in (with a squash option to avoid mixing in
>> the subproject's history).
>>
>> What about using this approach? That way Cython remains stable and
>> pinned on the right minivect version now and in the future, with no
>> burden on users.
>
> I still prefer having separate packages. I mean, we don't ship NumPy
> either, even though a lot of people use Cython together with it.

This is a very bad comparison. NumPy is not used by Cython at all, but 
by Cython-generated modules! Whereas minivect is a tool used in the 
compiler itself and working on the AST level.

Plex would be a better comparison (though bad as well, since Plex is not 
optional while minivect is).

> Keeping the two packages separate helps in keeping the interface between
> both clean. I wouldn't want to end up with Cython shipping some patched up
> version of minivect just because it's so easy, and I would like to allow
> users to install a new version of either Cython or minivect at any time.

I think this goal (allowing separate upgrades of Cython and/or minivect) 
is unrealistic and pointless.

I think you should look at minivect as "some AST transform algorithms 
which numba and Cython are able to share". It doesn't really have a life 
on its own, it's just a means for Cython and numba to cooperate. (Really 
long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, 
but that won't happen just yet. If it does, we can revisit this.)


Dag Sverre

From d.s.seljebotn at astro.uio.no  Sun Oct 14 10:23:01 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sun, 14 Oct 2012 10:23:01 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A755C.30802@astro.uio.no>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
Message-ID: <507A7665.8010601@astro.uio.no>

On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote:
> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>> mark florisson, 13.10.2012 20:30:
>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>
>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>> the master branch starts diverging all too far - it already
>>>>>>> requires a bit of adaptation.
>>>>>>> Did you manage to split off a separate minivect package?
>>>>>
>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>> interface than the implementation).
>>>>
>>>> Thanks for doing that, it'd be great to get this in (but myself I've
>>>> got
>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>> code and
>>>> the algorithms in minivect rather than the integration with Cython.
>>>>
>>>>> I don't see a reason for a new pull request.
>>>
>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>> subtree branch. Newer versions of git have a 'subtree' command
>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>> to split of, merge, push, and pull subdirectories.
>>>
>>> This means when users pull the master project, they get the
>>> sub-projects as well (without themselves needing newer git versions).
>>> Any changes to a subproject can be merged into the subproject, ands.
>>> changes can be pulled back in (with a squash option to avoid mixing in
>>> the subproject's history).
>>>
>>> What about using this approach? That way Cython remains stable and
>>> pinned on the right minivect version now and in the future, with no
>>> burden on users.
>>
>> I still prefer having separate packages. I mean, we don't ship NumPy
>> either, even though a lot of people use Cython together with it.
>
> This is a very bad comparison. NumPy is not used by Cython at all, but
> by Cython-generated modules! Whereas minivect is a tool used in the
> compiler itself and working on the AST level.
>
> Plex would be a better comparison (though bad as well, since Plex is not
> optional while minivect is).
>
>> Keeping the two packages separate helps in keeping the interface between
>> both clean. I wouldn't want to end up with Cython shipping some
>> patched up
>> version of minivect just because it's so easy, and I would like to allow
>> users to install a new version of either Cython or minivect at any time.
>
> I think this goal (allowing separate upgrades of Cython and/or minivect)
> is unrealistic and pointless.
>
> I think you should look at minivect as "some AST transform algorithms
> which numba and Cython are able to share". It doesn't really have a life
> on its own, it's just a means for Cython and numba to cooperate. (Really
> long-term then hopefully NumPy, numexpr, Theano etc. would jump on too,
> but that won't happen just yet. If it does, we can revisit this.)

Also, I don't even know when/if numba will use it... Mark, are you able 
to fill us in on that?

Dag Sverre

From stefan_ml at behnel.de  Sun Oct 14 10:32:00 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 10:32:00 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A755C.30802@astro.uio.no>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
Message-ID: <507A7880.7030501@behnel.de>

Dag Sverre Seljebotn, 14.10.2012 10:18:
> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>> mark florisson, 13.10.2012 20:30:
>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>
>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>> the master branch starts diverging all too far - it already
>>>>>>> requires a bit of adaptation.
>>>>>>> Did you manage to split off a separate minivect package?
>>>>>
>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>> interface than the implementation).
>>>>
>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>> code and
>>>> the algorithms in minivect rather than the integration with Cython.
>>>>
>>>>> I don't see a reason for a new pull request.
>>>
>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>> subtree branch. Newer versions of git have a 'subtree' command
>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>> to split of, merge, push, and pull subdirectories.
>>>
>>> This means when users pull the master project, they get the
>>> sub-projects as well (without themselves needing newer git versions).
>>> Any changes to a subproject can be merged into the subproject, ands.
>>> changes can be pulled back in (with a squash option to avoid mixing in
>>> the subproject's history).
>>>
>>> What about using this approach? That way Cython remains stable and
>>> pinned on the right minivect version now and in the future, with no
>>> burden on users.
>>
>> I still prefer having separate packages. I mean, we don't ship NumPy
>> either, even though a lot of people use Cython together with it.
> 
> This is a very bad comparison. NumPy is not used by Cython at all, but by
> Cython-generated modules! Whereas minivect is a tool used in the compiler
> itself and working on the AST level.
> 
> Plex would be a better comparison (though bad as well, since Plex is not
> optional while minivect is).
> 
>> Keeping the two packages separate helps in keeping the interface between
>> both clean. I wouldn't want to end up with Cython shipping some patched up
>> version of minivect just because it's so easy, and I would like to allow
>> users to install a new version of either Cython or minivect at any time.
> 
> I think this goal (allowing separate upgrades of Cython and/or minivect) is
> unrealistic and pointless.
> 
> I think you should look at minivect as "some AST transform algorithms which
> numba and Cython are able to share". It doesn't really have a life on its
> own, it's just a means for Cython and numba to cooperate. (Really long-term
> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
> won't happen just yet. If it does, we can revisit this.)

Ok, so I gave bad examples, fair enough. But I still don't see why we
should integrate minivect with Cython more deeply than necessary. Could you
explain what the advantage would be over having two separate packages?

Stefan


From d.s.seljebotn at astro.uio.no  Sun Oct 14 11:03:57 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sun, 14 Oct 2012 11:03:57 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A7880.7030501@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
Message-ID: <507A7FFD.7050408@astro.uio.no>

On 10/14/2012 10:32 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 14.10.2012 10:18:
>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>> mark florisson, 13.10.2012 20:30:
>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>
>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>> requires a bit of adaptation.
>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>
>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>> interface than the implementation).
>>>>>
>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>> code and
>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>
>>>>>> I don't see a reason for a new pull request.
>>>>
>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>> to split of, merge, push, and pull subdirectories.
>>>>
>>>> This means when users pull the master project, they get the
>>>> sub-projects as well (without themselves needing newer git versions).
>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>> the subproject's history).
>>>>
>>>> What about using this approach? That way Cython remains stable and
>>>> pinned on the right minivect version now and in the future, with no
>>>> burden on users.
>>>
>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>> either, even though a lot of people use Cython together with it.
>>
>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>> itself and working on the AST level.
>>
>> Plex would be a better comparison (though bad as well, since Plex is not
>> optional while minivect is).
>>
>>> Keeping the two packages separate helps in keeping the interface between
>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>> version of minivect just because it's so easy, and I would like to allow
>>> users to install a new version of either Cython or minivect at any time.
>>
>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>> unrealistic and pointless.
>>
>> I think you should look at minivect as "some AST transform algorithms which
>> numba and Cython are able to share". It doesn't really have a life on its
>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>> won't happen just yet. If it does, we can revisit this.)
>
> Ok, so I gave bad examples, fair enough. But I still don't see why we
> should integrate minivect with Cython more deeply than necessary. Could you
> explain what the advantage would be over having two separate packages?

Mark already raised the issue and we discussed in through in another 
thread where you participated, and me, Mark and Robert at least agreed 
on something very similar to what Mark is proposing to do now.

If you want to reopen the discussion I wish you had rather dug up the 
old thread again and respond to arguments in it, rather than us having 
to repeat the arguments here.

(Though please also consider what it does to discussion climate to 
reopen discussions about trivial details that can also trivially be 
changed later if better arguments crop up. Mark had a weekend of spare 
time to get this merged (yay!), and now who-knows-what happens because 
you decided to bikeshed something that had, at least to the eyes of rest 
of us, achieved consensus on this list.)

Dag







From stefan_ml at behnel.de  Sun Oct 14 12:23:29 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 12:23:29 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A7FFD.7050408@astro.uio.no>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de> <507A7FFD.7050408@astro.uio.no>
Message-ID: <507A92A1.1090304@behnel.de>

Dag Sverre Seljebotn, 14.10.2012 11:03:
> On 10/14/2012 10:32 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 14.10.2012 10:18:
>>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>> mark florisson, 13.10.2012 20:30:
>>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>>
>>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>>> requires a bit of adaptation.
>>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>>
>>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>>> interface than the implementation).
>>>>>>
>>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>>> code and
>>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>>
>>>>>>> I don't see a reason for a new pull request.
>>>>>
>>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>>> to split of, merge, push, and pull subdirectories.
>>>>>
>>>>> This means when users pull the master project, they get the
>>>>> sub-projects as well (without themselves needing newer git versions).
>>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>>> the subproject's history).
>>>>>
>>>>> What about using this approach? That way Cython remains stable and
>>>>> pinned on the right minivect version now and in the future, with no
>>>>> burden on users.
>>>>
>>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>>> either, even though a lot of people use Cython together with it.
>>>
>>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>>> itself and working on the AST level.
>>>
>>> Plex would be a better comparison (though bad as well, since Plex is not
>>> optional while minivect is).
>>>
>>>> Keeping the two packages separate helps in keeping the interface between
>>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>>> version of minivect just because it's so easy, and I would like to allow
>>>> users to install a new version of either Cython or minivect at any time.
>>>
>>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>>> unrealistic and pointless.
>>>
>>> I think you should look at minivect as "some AST transform algorithms which
>>> numba and Cython are able to share". It doesn't really have a life on its
>>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>>> won't happen just yet. If it does, we can revisit this.)
>>
>> Ok, so I gave bad examples, fair enough. But I still don't see why we
>> should integrate minivect with Cython more deeply than necessary. Could you
>> explain what the advantage would be over having two separate packages?
> 
> Mark already raised the issue and we discussed in through in another thread

This is still the same thread, at least for me. I replied to it when asking
for progress.


> where you participated, and me, Mark and Robert at least agreed on
> something very similar to what Mark is proposing to do now.
> 
> If you want to reopen the discussion I wish you had rather dug up the old
> thread again and respond to arguments in it, rather than us having to
> repeat the arguments here.

The single argument for git based integration was that it simplifies the
workflow during development. Meaning, it's a development-only thing and
deployments would use separate package installations.

I think that's ok as long as it's actually helpful for development and
doesn't start getting in the way. I can't comment on that yet, but I
wouldn't want to have to deal with it.


> (Though please also consider what it does to discussion climate to reopen
> discussions about trivial details that can also trivially be changed later
> if better arguments crop up.

It may be a trivial detail as long as it only has a reasonable impact on
the development. If it has a user impact, it's no longer immediately
obvious that it's trivial.


> Mark had a weekend of spare time to get this
> merged (yay!), and now who-knows-what happens because you decided to
> bikeshed something that had, at least to the eyes of rest of us, achieved
> consensus on this list.)

Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of
what he's doing now just wasn't clear enough to me.

Mark, could you explain in a bit more detail in which cases the git version
will be used and in which cases users would (or can) install minivect
separately?

Stefan


From d.s.seljebotn at astro.uio.no  Sun Oct 14 13:24:36 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sun, 14 Oct 2012 13:24:36 +0200
Subject: [Cython] array expressions
In-Reply-To: <507A92A1.1090304@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de> <507A7FFD.7050408@astro.uio.no>
	<507A92A1.1090304@behnel.de>
Message-ID: <507AA0F4.6080007@astro.uio.no>

On 10/14/2012 12:23 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 14.10.2012 11:03:
>> On 10/14/2012 10:32 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 14.10.2012 10:18:
>>>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>>> mark florisson, 13.10.2012 20:30:
>>>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>>>
>>>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>>>> requires a bit of adaptation.
>>>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>>>
>>>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>>>> interface than the implementation).
>>>>>>>
>>>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>>>> code and
>>>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>>>
>>>>>>>> I don't see a reason for a new pull request.
>>>>>>
>>>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>>>> to split of, merge, push, and pull subdirectories.
>>>>>>
>>>>>> This means when users pull the master project, they get the
>>>>>> sub-projects as well (without themselves needing newer git versions).
>>>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>>>> the subproject's history).
>>>>>>
>>>>>> What about using this approach? That way Cython remains stable and
>>>>>> pinned on the right minivect version now and in the future, with no
>>>>>> burden on users.
>>>>>
>>>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>>>> either, even though a lot of people use Cython together with it.
>>>>
>>>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>>>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>>>> itself and working on the AST level.
>>>>
>>>> Plex would be a better comparison (though bad as well, since Plex is not
>>>> optional while minivect is).
>>>>
>>>>> Keeping the two packages separate helps in keeping the interface between
>>>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>>>> version of minivect just because it's so easy, and I would like to allow
>>>>> users to install a new version of either Cython or minivect at any time.
>>>>
>>>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>>>> unrealistic and pointless.
>>>>
>>>> I think you should look at minivect as "some AST transform algorithms which
>>>> numba and Cython are able to share". It doesn't really have a life on its
>>>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>>>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>>>> won't happen just yet. If it does, we can revisit this.)
>>>
>>> Ok, so I gave bad examples, fair enough. But I still don't see why we
>>> should integrate minivect with Cython more deeply than necessary. Could you
>>> explain what the advantage would be over having two separate packages?
>>
>> Mark already raised the issue and we discussed in through in another thread
>
> This is still the same thread, at least for me. I replied to it when asking
> for progress.
>
>
>> where you participated, and me, Mark and Robert at least agreed on
>> something very similar to what Mark is proposing to do now.
>>
>> If you want to reopen the discussion I wish you had rather dug up the old
>> thread again and respond to arguments in it, rather than us having to
>> repeat the arguments here.
>
> The single argument for git based integration was that it simplifies the
> workflow during development. Meaning, it's a development-only thing and
> deployments would use separate package installations.
>
> I think that's ok as long as it's actually helpful for development and
> doesn't start getting in the way. I can't comment on that yet, but I
> wouldn't want to have to deal with it.

You know what -- I didn't think things through and was totally wrong here.

'git subtree' has a very different feel to it than 'git submodule'; it 
means that minivect will be in the tree by default and would have to be 
stripped out when making a release (or included by default).

So revisiting this discussion is indeed in order, as git subtree wasn't 
mentioned last time.

Dag Sverre

>
>
>> (Though please also consider what it does to discussion climate to reopen
>> discussions about trivial details that can also trivially be changed later
>> if better arguments crop up.
>
> It may be a trivial detail as long as it only has a reasonable impact on
> the development. If it has a user impact, it's no longer immediately
> obvious that it's trivial.
>
>
>> Mark had a weekend of spare time to get this
>> merged (yay!), and now who-knows-what happens because you decided to
>> bikeshed something that had, at least to the eyes of rest of us, achieved
>> consensus on this list.)
>
> Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of
> what he's doing now just wasn't clear enough to me.
>
> Mark, could you explain in a bit more detail in which cases the git version
> will be used and in which cases users would (or can) install minivect
> separately?
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>


From markflorisson88 at gmail.com  Sun Oct 14 13:59:04 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 14 Oct 2012 12:59:04 +0100
Subject: [Cython] array expressions
In-Reply-To: <507A7665.8010601@astro.uio.no>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7665.8010601@astro.uio.no>
Message-ID: <CANg26EVBqsFLE=hrjqhXVGNLYnb0cWLVSerAdwvopktMpNbZ+Q@mail.gmail.com>

On 14 October 2012 09:23, Dag Sverre Seljebotn
<d.s.seljebotn at astro.uio.no> wrote:
> On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote:
>>
>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>
>>> mark florisson, 13.10.2012 20:30:
>>>>
>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>
>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>
>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>
>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>
>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>
>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>
>>>>>>>>
>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>> requires a bit of adaptation.
>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>
>>>>>>
>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>> interface than the implementation).
>>>>>
>>>>>
>>>>> Thanks for doing that, it'd be great to get this in (but myself I've
>>>>> got
>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>> code and
>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>
>>>>>> I don't see a reason for a new pull request.
>>>>
>>>>
>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>> to split of, merge, push, and pull subdirectories.
>>>>
>>>> This means when users pull the master project, they get the
>>>> sub-projects as well (without themselves needing newer git versions).
>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>> the subproject's history).
>>>>
>>>> What about using this approach? That way Cython remains stable and
>>>> pinned on the right minivect version now and in the future, with no
>>>> burden on users.
>>>
>>>
>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>> either, even though a lot of people use Cython together with it.
>>
>>
>> This is a very bad comparison. NumPy is not used by Cython at all, but
>> by Cython-generated modules! Whereas minivect is a tool used in the
>> compiler itself and working on the AST level.
>>
>> Plex would be a better comparison (though bad as well, since Plex is not
>> optional while minivect is).
>>
>>> Keeping the two packages separate helps in keeping the interface between
>>> both clean. I wouldn't want to end up with Cython shipping some
>>> patched up
>>> version of minivect just because it's so easy, and I would like to allow
>>> users to install a new version of either Cython or minivect at any time.
>>
>>
>> I think this goal (allowing separate upgrades of Cython and/or minivect)
>> is unrealistic and pointless.
>>
>> I think you should look at minivect as "some AST transform algorithms
>> which numba and Cython are able to share". It doesn't really have a life
>> on its own, it's just a means for Cython and numba to cooperate. (Really
>> long-term then hopefully NumPy, numexpr, Theano etc. would jump on too,
>> but that won't happen just yet. If it does, we can revisit this.)
>
>
> Also, I don't even know when/if numba will use it... Mark, are you able to
> fill us in on that?
>

Yes, numba uses minivect, currently as a subtree (did that yesterday).
Submodules are very bad, especially if you develop the submodule
itself, and switch branches in the superproject (it will just forget
about unpinned changes, which means you just lose your work without so
much as a warning).

Numba mostly uses and extends the minivect type system and related
type conversions (to/from numpy dtypes, to/from ctypes, etc). NumbaPro
uses it for array expressions and ufuncs, which will at some point
make it into numba.

You're entirely right in saying that it breathes only through other
projects. I'll add the rest of the response in answer to Stefan's
question.

> Dag Sverre
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sun Oct 14 13:59:31 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 14 Oct 2012 12:59:31 +0100
Subject: [Cython] array expressions
In-Reply-To: <507A7880.7030501@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
Message-ID: <CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>

On 14 October 2012 09:32, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 14.10.2012 10:18:
>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>> mark florisson, 13.10.2012 20:30:
>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>
>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>> requires a bit of adaptation.
>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>
>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>> interface than the implementation).
>>>>>
>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>> code and
>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>
>>>>>> I don't see a reason for a new pull request.
>>>>
>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>> to split of, merge, push, and pull subdirectories.
>>>>
>>>> This means when users pull the master project, they get the
>>>> sub-projects as well (without themselves needing newer git versions).
>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>> the subproject's history).
>>>>
>>>> What about using this approach? That way Cython remains stable and
>>>> pinned on the right minivect version now and in the future, with no
>>>> burden on users.
>>>
>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>> either, even though a lot of people use Cython together with it.
>>
>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>> itself and working on the AST level.
>>
>> Plex would be a better comparison (though bad as well, since Plex is not
>> optional while minivect is).
>>
>>> Keeping the two packages separate helps in keeping the interface between
>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>> version of minivect just because it's so easy, and I would like to allow
>>> users to install a new version of either Cython or minivect at any time.
>>
>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>> unrealistic and pointless.
>>
>> I think you should look at minivect as "some AST transform algorithms which
>> numba and Cython are able to share". It doesn't really have a life on its
>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>> won't happen just yet. If it does, we can revisit this.)
>
> Ok, so I gave bad examples, fair enough. But I still don't see why we
> should integrate minivect with Cython more deeply than necessary. Could you
> explain what the advantage would be over having two separate packages?

The problem with minivect as a package is that it caters to different
projects, which have different requirements. Cython and minivect are
quite closely coupled, and any future change, or in the future any
older version may not have the functionality Cython needs, it's not
exactly a stable API at this point. For instance Numba needs python
2.7, whereas Cython needs to be compatible with python 2.4.

Before releasing minivect I'll verify every time that it doesn't break
Cython, but I currently have no real promises for backwards or forward
compatibility. And that is really because not all use cases have yet
been anticipated, and some really require a change, as I've already
seen with Numba.

We could list minivect as a dependency, which works for
easy_install/pip users, but I just foresee numerous people running
into problems that didn't install with pip, and I don't think an
exclusion of a 300kb addition is worth any of that.

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From markflorisson88 at gmail.com  Sun Oct 14 14:02:14 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 14 Oct 2012 13:02:14 +0100
Subject: [Cython] array expressions
In-Reply-To: <507A92A1.1090304@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de> <507A7FFD.7050408@astro.uio.no>
	<507A92A1.1090304@behnel.de>
Message-ID: <CANg26EXieri--NVQZqgzbcuC-=SQtvqGqLutzdTNBxAJfoObHg@mail.gmail.com>

On 14 October 2012 11:23, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 14.10.2012 11:03:
>> On 10/14/2012 10:32 AM, Stefan Behnel wrote:
>>> Dag Sverre Seljebotn, 14.10.2012 10:18:
>>>> On 10/14/2012 08:18 AM, Stefan Behnel wrote:
>>>>> mark florisson, 13.10.2012 20:30:
>>>>>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote:
>>>>>>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote:
>>>>>>>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote:
>>>>>>>>> On 12 October 2012 08:36, Stefan Behnel wrote:
>>>>>>>>>> mark florisson, 24.08.2012 20:40:
>>>>>>>>>>> Here a pull request for element-wise array expressions for Cython:
>>>>>>>>>>> https://github.com/cython/cython/pull/144
>>>>>>>>>>
>>>>>>>>>> Mark, any news on this? I'd like to see a version merged before
>>>>>>>>>> the master branch starts diverging all too far - it already
>>>>>>>>>> requires a bit of adaptation.
>>>>>>>>>> Did you manage to split off a separate minivect package?
>>>>>>>>
>>>>>>>> I'm assuming this has already been looked at, at least to some level,
>>>>>>>> by Dag, but I'll try to take a brief pass at it too (probably more the
>>>>>>>> interface than the implementation).
>>>>>>>
>>>>>>> Thanks for doing that, it'd be great to get this in (but myself I've got
>>>>>>> nothing to spare). I'll admit I was mostly focused on the generated
>>>>>>> code and
>>>>>>> the algorithms in minivect rather than the integration with Cython.
>>>>>>>
>>>>>>>> I don't see a reason for a new pull request.
>>>>>>
>>>>>> Great. As for the packaging, I'm creating a distribution branch, and a
>>>>>> subtree branch. Newer versions of git have a 'subtree' command
>>>>>> (previously https://github.com/apenwarr/git-subtree), which allows one
>>>>>> to split of, merge, push, and pull subdirectories.
>>>>>>
>>>>>> This means when users pull the master project, they get the
>>>>>> sub-projects as well (without themselves needing newer git versions).
>>>>>> Any changes to a subproject can be merged into the subproject, ands.
>>>>>> changes can be pulled back in (with a squash option to avoid mixing in
>>>>>> the subproject's history).
>>>>>>
>>>>>> What about using this approach? That way Cython remains stable and
>>>>>> pinned on the right minivect version now and in the future, with no
>>>>>> burden on users.
>>>>>
>>>>> I still prefer having separate packages. I mean, we don't ship NumPy
>>>>> either, even though a lot of people use Cython together with it.
>>>>
>>>> This is a very bad comparison. NumPy is not used by Cython at all, but by
>>>> Cython-generated modules! Whereas minivect is a tool used in the compiler
>>>> itself and working on the AST level.
>>>>
>>>> Plex would be a better comparison (though bad as well, since Plex is not
>>>> optional while minivect is).
>>>>
>>>>> Keeping the two packages separate helps in keeping the interface between
>>>>> both clean. I wouldn't want to end up with Cython shipping some patched up
>>>>> version of minivect just because it's so easy, and I would like to allow
>>>>> users to install a new version of either Cython or minivect at any time.
>>>>
>>>> I think this goal (allowing separate upgrades of Cython and/or minivect) is
>>>> unrealistic and pointless.
>>>>
>>>> I think you should look at minivect as "some AST transform algorithms which
>>>> numba and Cython are able to share". It doesn't really have a life on its
>>>> own, it's just a means for Cython and numba to cooperate. (Really long-term
>>>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that
>>>> won't happen just yet. If it does, we can revisit this.)
>>>
>>> Ok, so I gave bad examples, fair enough. But I still don't see why we
>>> should integrate minivect with Cython more deeply than necessary. Could you
>>> explain what the advantage would be over having two separate packages?
>>
>> Mark already raised the issue and we discussed in through in another thread
>
> This is still the same thread, at least for me. I replied to it when asking
> for progress.
>
>
>> where you participated, and me, Mark and Robert at least agreed on
>> something very similar to what Mark is proposing to do now.
>>
>> If you want to reopen the discussion I wish you had rather dug up the old
>> thread again and respond to arguments in it, rather than us having to
>> repeat the arguments here.
>
> The single argument for git based integration was that it simplifies the
> workflow during development. Meaning, it's a development-only thing and
> deployments would use separate package installations.
>
> I think that's ok as long as it's actually helpful for development and
> doesn't start getting in the way. I can't comment on that yet, but I
> wouldn't want to have to deal with it.
>
>
>> (Though please also consider what it does to discussion climate to reopen
>> discussions about trivial details that can also trivially be changed later
>> if better arguments crop up.
>
> It may be a trivial detail as long as it only has a reasonable impact on
> the development. If it has a user impact, it's no longer immediately
> obvious that it's trivial.
>
>
>> Mark had a weekend of spare time to get this
>> merged (yay!), and now who-knows-what happens because you decided to
>> bikeshed something that had, at least to the eyes of rest of us, achieved
>> consensus on this list.)
>
> Maybe the "consensus" wasn't clear enough, or maybe Mark's explanation of
> what he's doing now just wasn't clear enough to me.
>
> Mark, could you explain in a bit more detail in which cases the git version
> will be used and in which cases users would (or can) install minivect
> separately?

The subtree plan is to always use the included version, which is also
automatic for a git checkout. The dependency plan is to support
pip/easy_install, pinned on a specific minivect release.

Another problem with B) arises when minivect is included in other
projects as a dependency, and the projects have conflicting minivect
version requirements.

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sun Oct 14 15:05:07 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 15:05:07 +0200
Subject: [Cython] array expressions
In-Reply-To: <CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
Message-ID: <507AB883.2020109@behnel.de>

mark florisson, 14.10.2012 13:59:
> The problem with minivect as a package is that it caters to different
> projects, which have different requirements. Cython and minivect are
> quite closely coupled, and any future change, or in the future any
> older version may not have the functionality Cython needs, it's not
> exactly a stable API at this point.

Ok, understood.


> For instance Numba needs python
> 2.7, whereas Cython needs to be compatible with python 2.4.
> 
> Before releasing minivect I'll verify every time that it doesn't break
> Cython, but I currently have no real promises for backwards or forward
> compatibility. And that is really because not all use cases have yet
> been anticipated, and some really require a change, as I've already
> seen with Numba.
> 
> We could list minivect as a dependency, which works for
> easy_install/pip users, but I just foresee numerous people running
> into problems that didn't install with pip, and I don't think an
> exclusion of a 300kb addition is worth any of that.

Fine. In that case, I'm for not making minivect a separate package at all
but including it directly and considering it a part of Cython (and Numba
etc.) until there is enough of an interface to make it a reusable separate
package, or at least to support a separate installation and independent
update. Basically, if you can't update it separately, there's no use in
installing it separately.

As long as we handle this so, we should take care to keep the generic parts
in their separate package directory and the Cython specific parts in
Cython, and try to keep the interface between the two as cleanly separate
as possible, so that we can actually reach a point where both have an
interface. I would guess that the need to support Numba from the same
source base will encourage this kind of separation anyway.

Note that this means that minivect will fall under the release schedules of
Cython and Numba (independently), instead of really having its own releases.

BTW, I would guess that no-one has tested hg with git subtrees yet?

Stefan


From markflorisson88 at gmail.com  Sun Oct 14 15:13:28 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 14 Oct 2012 14:13:28 +0100
Subject: [Cython] array expressions
In-Reply-To: <507AB883.2020109@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
Message-ID: <CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>

On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 14.10.2012 13:59:
>> The problem with minivect as a package is that it caters to different
>> projects, which have different requirements. Cython and minivect are
>> quite closely coupled, and any future change, or in the future any
>> older version may not have the functionality Cython needs, it's not
>> exactly a stable API at this point.
>
> Ok, understood.
>
>
>> For instance Numba needs python
>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>
>> Before releasing minivect I'll verify every time that it doesn't break
>> Cython, but I currently have no real promises for backwards or forward
>> compatibility. And that is really because not all use cases have yet
>> been anticipated, and some really require a change, as I've already
>> seen with Numba.
>>
>> We could list minivect as a dependency, which works for
>> easy_install/pip users, but I just foresee numerous people running
>> into problems that didn't install with pip, and I don't think an
>> exclusion of a 300kb addition is worth any of that.
>
> Fine. In that case, I'm for not making minivect a separate package at all
> but including it directly and considering it a part of Cython (and Numba
> etc.) until there is enough of an interface to make it a reusable separate
> package, or at least to support a separate installation and independent
> update. Basically, if you can't update it separately, there's no use in
> installing it separately.
>
> As long as we handle this so, we should take care to keep the generic parts
> in their separate package directory and the Cython specific parts in
> Cython, and try to keep the interface between the two as cleanly separate
> as possible, so that we can actually reach a point where both have an
> interface. I would guess that the need to support Numba from the same
> source base will encourage this kind of separation anyway.

Yes, definitely.

> Note that this means that minivect will fall under the release schedules of
> Cython and Numba (independently), instead of really having its own releases.

It can have its own releases as well, but currently there isn't much
point :) Minivect can be developed independent of the releases, since
Cython and Numba need to explicitly pull in the changes. Let's make a
habit of squashing the minivect pulls to avoid its history.

I'll also wait for Dag and Robert to see if they have a (final)
opinion before merging the subtree.

> BTW, I would guess that no-one has tested hg with git subtrees yet?

Heh, no. The hg support didn't work great for me (it was really slow),
can I ask why you're using it?

> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

From stefan_ml at behnel.de  Sun Oct 14 15:39:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Oct 2012 15:39:28 +0200
Subject: [Cython] array expressions
In-Reply-To: <CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
Message-ID: <507AC090.6000907@behnel.de>

mark florisson, 14.10.2012 15:13:
> The hg support didn't work great for me (it was really slow),

It's fast enough for almost everything. What's really slow is when it has
to figure out how to match branches after head changes - I consider that a
pretty serious bug. But that happens quite rarely. And I usually push only
the master branch anyway, which is quick.

Obviously, hg is way faster against a mercurial server than against a git
server.


> can I ask why you're using it?

I prefer the interface of hg over that of git, that's the main reason.

In fact, whenever I have to touch git, it quickly lives up to its name:

http://dict.leo.org/ende?lang=en&search=igitt

Stefan


From d.s.seljebotn at astro.uio.no  Sun Oct 14 16:27:47 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sun, 14 Oct 2012 16:27:47 +0200
Subject: [Cython] array expressions
In-Reply-To: <507AB883.2020109@behnel.de>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
Message-ID: <507ACBE3.3020302@astro.uio.no>

On 10/14/2012 03:05 PM, Stefan Behnel wrote:
> mark florisson, 14.10.2012 13:59:
>> The problem with minivect as a package is that it caters to different
>> projects, which have different requirements. Cython and minivect are
>> quite closely coupled, and any future change, or in the future any
>> older version may not have the functionality Cython needs, it's not
>> exactly a stable API at this point.
>
> Ok, understood.
>
>
>> For instance Numba needs python
>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>
>> Before releasing minivect I'll verify every time that it doesn't break
>> Cython, but I currently have no real promises for backwards or forward
>> compatibility. And that is really because not all use cases have yet
>> been anticipated, and some really require a change, as I've already
>> seen with Numba.
>>
>> We could list minivect as a dependency, which works for
>> easy_install/pip users, but I just foresee numerous people running
>> into problems that didn't install with pip, and I don't think an
>> exclusion of a 300kb addition is worth any of that.
>
> Fine. In that case, I'm for not making minivect a separate package at all
> but including it directly and considering it a part of Cython (and Numba
> etc.) until there is enough of an interface to make it a reusable separate
> package, or at least to support a separate installation and independent
> update. Basically, if you can't update it separately, there's no use in
> installing it separately.
>
> As long as we handle this so, we should take care to keep the generic parts
> in their separate package directory and the Cython specific parts in
> Cython, and try to keep the interface between the two as cleanly separate
> as possible, so that we can actually reach a point where both have an
> interface. I would guess that the need to support Numba from the same
> source base will encourage this kind of separation anyway.
>
> Note that this means that minivect will fall under the release schedules of
> Cython and Numba (independently), instead of really having its own releases.

+1 to Mark's proposal, this seems nice.

> BTW, I would guess that no-one has tested hg with git subtrees yet?

git subtree is just a utility for pushing/pulling the history of a 
subdirectory to/from another repository.

As long as you don't need to update to a new version of minivect, you 
shouldn't be affected at all; when you don't actually use the "git 
subtree" commands, it's just a normal directory under git control.

But if one needs to fix a bug in minivect when doing Cython development, 
I'm guessing hg-git can't help you move the patch upstream (but you can 
always just submit a normal Unix diff upstream instead if you don't want 
to use git).

Dag Sverre

From robertwb at gmail.com  Tue Oct 16 19:48:18 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Tue, 16 Oct 2012 10:48:18 -0700
Subject: [Cython] array expressions
In-Reply-To: <CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
Message-ID: <CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>

On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 14.10.2012 13:59:
>>> The problem with minivect as a package is that it caters to different
>>> projects, which have different requirements. Cython and minivect are
>>> quite closely coupled, and any future change, or in the future any
>>> older version may not have the functionality Cython needs, it's not
>>> exactly a stable API at this point.
>>
>> Ok, understood.
>>
>>
>>> For instance Numba needs python
>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>
>>> Before releasing minivect I'll verify every time that it doesn't break
>>> Cython, but I currently have no real promises for backwards or forward
>>> compatibility. And that is really because not all use cases have yet
>>> been anticipated, and some really require a change, as I've already
>>> seen with Numba.
>>>
>>> We could list minivect as a dependency, which works for
>>> easy_install/pip users, but I just foresee numerous people running
>>> into problems that didn't install with pip, and I don't think an
>>> exclusion of a 300kb addition is worth any of that.
>>
>> Fine. In that case, I'm for not making minivect a separate package at all
>> but including it directly and considering it a part of Cython (and Numba
>> etc.) until there is enough of an interface to make it a reusable separate
>> package, or at least to support a separate installation and independent
>> update. Basically, if you can't update it separately, there's no use in
>> installing it separately.
>>
>> As long as we handle this so, we should take care to keep the generic parts
>> in their separate package directory and the Cython specific parts in
>> Cython, and try to keep the interface between the two as cleanly separate
>> as possible, so that we can actually reach a point where both have an
>> interface. I would guess that the need to support Numba from the same
>> source base will encourage this kind of separation anyway.
>
> Yes, definitely.
>
>> Note that this means that minivect will fall under the release schedules of
>> Cython and Numba (independently), instead of really having its own releases.
>
> It can have its own releases as well, but currently there isn't much
> point :) Minivect can be developed independent of the releases, since
> Cython and Numba need to explicitly pull in the changes. Let's make a
> habit of squashing the minivect pulls to avoid its history.
>
> I'll also wait for Dag and Robert to see if they have a (final)
> opinion before merging the subtree.

As I mentioned on the pull request, looks good to me. Given the
(somewhat) tight coupling with the AST but the desire to use the
codebase for multiple projects, a subtree seems to make the most sense
(until/if we have some kind of a plugin system).

- Robert

From markflorisson88 at gmail.com  Thu Oct 18 12:56:54 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 18 Oct 2012 11:56:54 +0100
Subject: [Cython] mypy
Message-ID: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>

Has anyone looked at this yet?

http://www.mypy-lang.org/index.html

From stefan_ml at behnel.de  Thu Oct 18 18:31:36 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Oct 2012 18:31:36 +0200
Subject: [Cython] mypy
In-Reply-To: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>
References: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>
Message-ID: <50802EE8.5010803@behnel.de>

mark florisson, 18.10.2012 12:56:
> Has anyone looked at this yet?
> http://www.mypy-lang.org/index.html

>From a quick glance, they use future tense a lot on their site. It's easy
to state what you would like to have at some point. We all know that it's
much harder to make it a reality, and then to make it useful on top of that.

Stefan


From vallee.aurelien at gmail.com  Thu Oct 18 18:37:21 2012
From: vallee.aurelien at gmail.com (=?ISO-8859-1?Q?Aur=E9lien_Vall=E9e?=)
Date: Thu, 18 Oct 2012 16:37:21 +0000
Subject: [Cython] mypy
In-Reply-To: <50802EE8.5010803@behnel.de>
References: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>
	<50802EE8.5010803@behnel.de>
Message-ID: <CAFVgj0ZhPkzipyXSDEeDPeb5N75h_qf4BzNsPRpiJW=ZZEjSUw@mail.gmail.com>

Same observation for me:
- A single developer,
- no community,
- no release.

The project needs some time to be considered.
Will recheck it in like 1 or 2 years.

On Thu, Oct 18, 2012 at 4:31 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> mark florisson, 18.10.2012 12:56:
> > Has anyone looked at this yet?
> > http://www.mypy-lang.org/index.html
>
> From a quick glance, they use future tense a lot on their site. It's easy
> to state what you would like to have at some point. We all know that it's
> much harder to make it a reality, and then to make it useful on top of
> that.
>
> Stefan
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>



-- 
Aur?lien Vall?e
+33 6 47 41 70 37
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/cython-devel/attachments/20121018/b3e66f3b/attachment.html>

From robin at reportlab.com  Thu Oct 18 18:39:49 2012
From: robin at reportlab.com (Robin Becker)
Date: Thu, 18 Oct 2012 17:39:49 +0100
Subject: [Cython] mypy
In-Reply-To: <50802EE8.5010803@behnel.de>
References: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>
	<50802EE8.5010803@behnel.de>
Message-ID: <508030D5.4000900@chamonix.reportlab.co.uk>

On 18/10/2012 17:31, Stefan Behnel wrote:
> mark florisson, 18.10.2012 12:56:
>> Has anyone looked at this yet?
>> http://www.mypy-lang.org/index.html
>
>>From a quick glance, they use future tense a lot on their site. It's easy
> to state what you would like to have at some point. We all know that it's
> much harder to make it a reality, and then to make it useful on top of that.
>
> Stefan
.....

there does seem to be a pre-existing language called "alore" with some of the 
features claimed. Of course I haven't even downloaded it yet so my expertise is 
limited. It claims native thread support etc etc.
-- 
Robin Becker

From robertwb at gmail.com  Fri Oct 19 01:15:21 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Thu, 18 Oct 2012 16:15:21 -0700
Subject: [Cython] mypy
In-Reply-To: <50802EE8.5010803@behnel.de>
References: <CANg26EVSrN4Dud-5oYD5aigGUy8toNkL41C3VL5mqeLjz20Uwg@mail.gmail.com>
	<50802EE8.5010803@behnel.de>
Message-ID: <CADiQ+QDcz9y1j3FTe36kVvzt70zmG2SsuOYVZWTKYmEmDJ39Uw@mail.gmail.com>

On Thu, Oct 18, 2012 at 9:31 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> mark florisson, 18.10.2012 12:56:
>> Has anyone looked at this yet?
>> http://www.mypy-lang.org/index.html
>
> From a quick glance, they use future tense a lot on their site. It's easy
> to state what you would like to have at some point. We all know that it's
> much harder to make it a reality, and then to make it useful on top of that.

That was exactly my first impression as well. There is some precedent
with his previous project, Alore, but no third-party analysis (or
use?) of that either.

He is aware of Cython: http://www.mypy-lang.org/faq.html#cython

- Robert

From markflorisson88 at gmail.com  Sun Oct 21 14:39:52 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 21 Oct 2012 13:39:52 +0100
Subject: [Cython] array expressions
In-Reply-To: <CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
	<CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
Message-ID: <CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>

On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> mark florisson, 14.10.2012 13:59:
>>>> The problem with minivect as a package is that it caters to different
>>>> projects, which have different requirements. Cython and minivect are
>>>> quite closely coupled, and any future change, or in the future any
>>>> older version may not have the functionality Cython needs, it's not
>>>> exactly a stable API at this point.
>>>
>>> Ok, understood.
>>>
>>>
>>>> For instance Numba needs python
>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>
>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>> Cython, but I currently have no real promises for backwards or forward
>>>> compatibility. And that is really because not all use cases have yet
>>>> been anticipated, and some really require a change, as I've already
>>>> seen with Numba.
>>>>
>>>> We could list minivect as a dependency, which works for
>>>> easy_install/pip users, but I just foresee numerous people running
>>>> into problems that didn't install with pip, and I don't think an
>>>> exclusion of a 300kb addition is worth any of that.
>>>
>>> Fine. In that case, I'm for not making minivect a separate package at all
>>> but including it directly and considering it a part of Cython (and Numba
>>> etc.) until there is enough of an interface to make it a reusable separate
>>> package, or at least to support a separate installation and independent
>>> update. Basically, if you can't update it separately, there's no use in
>>> installing it separately.
>>>
>>> As long as we handle this so, we should take care to keep the generic parts
>>> in their separate package directory and the Cython specific parts in
>>> Cython, and try to keep the interface between the two as cleanly separate
>>> as possible, so that we can actually reach a point where both have an
>>> interface. I would guess that the need to support Numba from the same
>>> source base will encourage this kind of separation anyway.
>>
>> Yes, definitely.
>>
>>> Note that this means that minivect will fall under the release schedules of
>>> Cython and Numba (independently), instead of really having its own releases.
>>
>> It can have its own releases as well, but currently there isn't much
>> point :) Minivect can be developed independent of the releases, since
>> Cython and Numba need to explicitly pull in the changes. Let's make a
>> habit of squashing the minivect pulls to avoid its history.
>>
>> I'll also wait for Dag and Robert to see if they have a (final)
>> opinion before merging the subtree.
>
> As I mentioned on the pull request, looks good to me. Given the
> (somewhat) tight coupling with the AST but the desire to use the
> codebase for multiple projects, a subtree seems to make the most sense
> (until/if we have some kind of a plugin system).
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Great, thanks for the review Robert. There seems to be a refcount
issue with python 3:

cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
/jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
[GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Cython.Compiler import Main
[98632 refs]
>>> ^D
[98632 refs]
python: Modules/gcmodule.c:327: visit_decref: Assertion
`gc->gc.gc_refs != 0' failed.

The object being visited is a UtilityCode object. I think the error
means the object is being visited more often than it's refcount value,
which means it's not increffed properly somewhere. I'm not sure how/if
my changes introduced this, has this problem been encountered recently
in Cython's master?

From markflorisson88 at gmail.com  Sun Oct 21 14:44:20 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 21 Oct 2012 13:44:20 +0100
Subject: [Cython] array expressions
In-Reply-To: <CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
	<CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
	<CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
Message-ID: <CANg26EWMMCb4_AUi9xPiX9G8TtWDB9314ZtrcMzLivrMM_DBJA@mail.gmail.com>

On 21 October 2012 13:39, mark florisson <markflorisson88 at gmail.com> wrote:
> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 14.10.2012 13:59:
>>>>> The problem with minivect as a package is that it caters to different
>>>>> projects, which have different requirements. Cython and minivect are
>>>>> quite closely coupled, and any future change, or in the future any
>>>>> older version may not have the functionality Cython needs, it's not
>>>>> exactly a stable API at this point.
>>>>
>>>> Ok, understood.
>>>>
>>>>
>>>>> For instance Numba needs python
>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>
>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>> compatibility. And that is really because not all use cases have yet
>>>>> been anticipated, and some really require a change, as I've already
>>>>> seen with Numba.
>>>>>
>>>>> We could list minivect as a dependency, which works for
>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>> into problems that didn't install with pip, and I don't think an
>>>>> exclusion of a 300kb addition is worth any of that.
>>>>
>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>> but including it directly and considering it a part of Cython (and Numba
>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>> package, or at least to support a separate installation and independent
>>>> update. Basically, if you can't update it separately, there's no use in
>>>> installing it separately.
>>>>
>>>> As long as we handle this so, we should take care to keep the generic parts
>>>> in their separate package directory and the Cython specific parts in
>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>> as possible, so that we can actually reach a point where both have an
>>>> interface. I would guess that the need to support Numba from the same
>>>> source base will encourage this kind of separation anyway.
>>>
>>> Yes, definitely.
>>>
>>>> Note that this means that minivect will fall under the release schedules of
>>>> Cython and Numba (independently), instead of really having its own releases.
>>>
>>> It can have its own releases as well, but currently there isn't much
>>> point :) Minivect can be developed independent of the releases, since
>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>> habit of squashing the minivect pulls to avoid its history.
>>>
>>> I'll also wait for Dag and Robert to see if they have a (final)
>>> opinion before merging the subtree.
>>
>> As I mentioned on the pull request, looks good to me. Given the
>> (somewhat) tight coupling with the AST but the desire to use the
>> codebase for multiple projects, a subtree seems to make the most sense
>> (until/if we have some kind of a plugin system).
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Great, thanks for the review Robert. There seems to be a refcount
> issue with python 3:
>
> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from Cython.Compiler import Main
> [98632 refs]
>>>> ^D
> [98632 refs]
> python: Modules/gcmodule.c:327: visit_decref: Assertion
> `gc->gc.gc_refs != 0' failed.
>
> The object being visited is a UtilityCode object. I think the error
> means the object is being visited more often than it's refcount value,
> which means it's not increffed properly somewhere. I'm not sure how/if
> my changes introduced this, has this problem been encountered recently
> in Cython's master?

This is running with -m trace --trace:

<snip>
Builtin.py(396):     for bf in builtin_function_table:
Builtin.py(397):         bf.declare_in_scope(builtin_scope)
 --- modulename: Builtin, funcname: declare_in_scope
Builtin.py(163):         func_type, sig = self.func_type, self.sig
Builtin.py(164):         if func_type is None:
Builtin.py(165):             if sig is None:
Builtin.py(166):                 sig = Signature(self.args, self.ret_type)
 --- modulename: TypeSlots, funcname: __init__
TypeSlots.py(85):         self.has_dummy_arg = 0
TypeSlots.py(86):         self.has_generic_args = 0
TypeSlots.py(87):         if arg_format[:1] == '-':
TypeSlots.py(90):         if arg_format[-1:] == '*':
TypeSlots.py(93):         self.fixed_arg_format = arg_format
TypeSlots.py(94):         self.ret_format = ret_format
TypeSlots.py(95):         self.error_value =
self.error_value_map.get(ret_format, None)
TypeSlots.py(96):         self.is_staticmethod = False
Builtin.py(167):             func_type = sig.function_type()
 --- modulename: TypeSlots, funcname: function_type
TypeSlots.py(125):         args = []
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
 --- modulename: TypeSlots, funcname: num_fixed_args
TypeSlots.py(99):         return len(self.fixed_arg_format)
TypeSlots.py(127):             if self_arg_override is not None and
self.is_self_arg(i):
TypeSlots.py(131):                 arg_type = self.fixed_arg_type(i)
 --- modulename: TypeSlots, funcname: fixed_arg_type
TypeSlots.py(110):         return self.format_map[self.fixed_arg_format[i]]
TypeSlots.py(132):
args.append(PyrexTypes.CFuncTypeArg("", arg_type, None))
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2966):         self.name = name
PyrexTypes.py(2967):         if cname is not None:
PyrexTypes.py(2970):             self.cname = Naming.var_prefix + name
PyrexTypes.py(2971):         self.type = type
PyrexTypes.py(2972):         self.pos = pos
PyrexTypes.py(2973):         self.needs_type_test = False # TODO:
should these defaults be set in analyse_types()?
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
TypeSlots.py(127):             if self_arg_override is not None and
self.is_self_arg(i):
TypeSlots.py(131):                 arg_type = self.fixed_arg_type(i)
 --- modulename: TypeSlots, funcname: fixed_arg_type
TypeSlots.py(110):         return self.format_map[self.fixed_arg_format[i]]
TypeSlots.py(132):
args.append(PyrexTypes.CFuncTypeArg("", arg_type, None))
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2966):         self.name = name
PyrexTypes.py(2967):         if cname is not None:
PyrexTypes.py(2970):             self.cname = Naming.var_prefix + name
PyrexTypes.py(2971):         self.type = type
PyrexTypes.py(2972):         self.pos = pos
PyrexTypes.py(2973):         self.needs_type_test = False # TODO:
should these defaults be set in analyse_types()?
TypeSlots.py(126):         for i in range(self.num_fixed_args()):
TypeSlots.py(133):         if self_arg_override is not None and
self.returns_self_type():
TypeSlots.py(136):             ret_type = self.return_type()
 --- modulename: TypeSlots, funcname: return_type
TypeSlots.py(113):         return self.format_map[self.ret_format]
TypeSlots.py(137):         exc_value = self.exception_value()
 --- modulename: TypeSlots, funcname: exception_value
TypeSlots.py(121):         return self.error_value_map.get(self.ret_format)
TypeSlots.py(138):         return PyrexTypes.CFuncType(ret_type, args,
exception_value = exc_value)
 --- modulename: PyrexTypes, funcname: __init__
PyrexTypes.py(2531):         self.return_type = return_type
python: Modules/gcmodule.c:327: visit_decref: Assertion
`gc->gc.gc_refs != 0' failed.
Aborted

Maybe I'll try rebasing on the latest master again.

From robertwb at gmail.com  Tue Oct 23 03:44:56 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 22 Oct 2012 18:44:56 -0700
Subject: [Cython] array expressions
In-Reply-To: <CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
	<CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
	<CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
Message-ID: <CADiQ+QAu28O45oBQgp65EkNRnzVRhYwQwVxe8e6j_s58CCQ6Aw@mail.gmail.com>

On Sun, Oct 21, 2012 at 5:39 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> mark florisson, 14.10.2012 13:59:
>>>>> The problem with minivect as a package is that it caters to different
>>>>> projects, which have different requirements. Cython and minivect are
>>>>> quite closely coupled, and any future change, or in the future any
>>>>> older version may not have the functionality Cython needs, it's not
>>>>> exactly a stable API at this point.
>>>>
>>>> Ok, understood.
>>>>
>>>>
>>>>> For instance Numba needs python
>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>
>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>> compatibility. And that is really because not all use cases have yet
>>>>> been anticipated, and some really require a change, as I've already
>>>>> seen with Numba.
>>>>>
>>>>> We could list minivect as a dependency, which works for
>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>> into problems that didn't install with pip, and I don't think an
>>>>> exclusion of a 300kb addition is worth any of that.
>>>>
>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>> but including it directly and considering it a part of Cython (and Numba
>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>> package, or at least to support a separate installation and independent
>>>> update. Basically, if you can't update it separately, there's no use in
>>>> installing it separately.
>>>>
>>>> As long as we handle this so, we should take care to keep the generic parts
>>>> in their separate package directory and the Cython specific parts in
>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>> as possible, so that we can actually reach a point where both have an
>>>> interface. I would guess that the need to support Numba from the same
>>>> source base will encourage this kind of separation anyway.
>>>
>>> Yes, definitely.
>>>
>>>> Note that this means that minivect will fall under the release schedules of
>>>> Cython and Numba (independently), instead of really having its own releases.
>>>
>>> It can have its own releases as well, but currently there isn't much
>>> point :) Minivect can be developed independent of the releases, since
>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>> habit of squashing the minivect pulls to avoid its history.
>>>
>>> I'll also wait for Dag and Robert to see if they have a (final)
>>> opinion before merging the subtree.
>>
>> As I mentioned on the pull request, looks good to me. Given the
>> (somewhat) tight coupling with the AST but the desire to use the
>> codebase for multiple projects, a subtree seems to make the most sense
>> (until/if we have some kind of a plugin system).
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Great, thanks for the review Robert. There seems to be a refcount
> issue with python 3:
>
> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> from Cython.Compiler import Main
> [98632 refs]
>>>> ^D
> [98632 refs]
> python: Modules/gcmodule.c:327: visit_decref: Assertion
> `gc->gc.gc_refs != 0' failed.
>
> The object being visited is a UtilityCode object. I think the error
> means the object is being visited more often than it's refcount value,
> which means it's not increffed properly somewhere. I'm not sure how/if
> my changes introduced this, has this problem been encountered recently
> in Cython's master?

Not that I'm aware of. The refnanny is supposed to catch this kind of
thing, is it enabled?

- Robert

From markflorisson88 at gmail.com  Tue Oct 23 11:47:48 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 23 Oct 2012 10:47:48 +0100
Subject: [Cython] array expressions
In-Reply-To: <CADiQ+QAu28O45oBQgp65EkNRnzVRhYwQwVxe8e6j_s58CCQ6Aw@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
	<CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
	<CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
	<CADiQ+QAu28O45oBQgp65EkNRnzVRhYwQwVxe8e6j_s58CCQ6Aw@mail.gmail.com>
Message-ID: <CANg26EUcO0LJXT+B-hd9eS_XymMmG72bKYmZDGU6W7sT_C0DEw@mail.gmail.com>

On 23 October 2012 02:44, Robert Bradshaw <robertwb at gmail.com> wrote:
> On Sun, Oct 21, 2012 at 5:39 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>> mark florisson, 14.10.2012 13:59:
>>>>>> The problem with minivect as a package is that it caters to different
>>>>>> projects, which have different requirements. Cython and minivect are
>>>>>> quite closely coupled, and any future change, or in the future any
>>>>>> older version may not have the functionality Cython needs, it's not
>>>>>> exactly a stable API at this point.
>>>>>
>>>>> Ok, understood.
>>>>>
>>>>>
>>>>>> For instance Numba needs python
>>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>>
>>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>>> compatibility. And that is really because not all use cases have yet
>>>>>> been anticipated, and some really require a change, as I've already
>>>>>> seen with Numba.
>>>>>>
>>>>>> We could list minivect as a dependency, which works for
>>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>>> into problems that didn't install with pip, and I don't think an
>>>>>> exclusion of a 300kb addition is worth any of that.
>>>>>
>>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>>> but including it directly and considering it a part of Cython (and Numba
>>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>>> package, or at least to support a separate installation and independent
>>>>> update. Basically, if you can't update it separately, there's no use in
>>>>> installing it separately.
>>>>>
>>>>> As long as we handle this so, we should take care to keep the generic parts
>>>>> in their separate package directory and the Cython specific parts in
>>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>>> as possible, so that we can actually reach a point where both have an
>>>>> interface. I would guess that the need to support Numba from the same
>>>>> source base will encourage this kind of separation anyway.
>>>>
>>>> Yes, definitely.
>>>>
>>>>> Note that this means that minivect will fall under the release schedules of
>>>>> Cython and Numba (independently), instead of really having its own releases.
>>>>
>>>> It can have its own releases as well, but currently there isn't much
>>>> point :) Minivect can be developed independent of the releases, since
>>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>>> habit of squashing the minivect pulls to avoid its history.
>>>>
>>>> I'll also wait for Dag and Robert to see if they have a (final)
>>>> opinion before merging the subtree.
>>>
>>> As I mentioned on the pull request, looks good to me. Given the
>>> (somewhat) tight coupling with the AST but the desire to use the
>>> codebase for multiple projects, a subtree seems to make the most sense
>>> (until/if we have some kind of a plugin system).
>>>
>>> - Robert
>>> _______________________________________________
>>> cython-devel mailing list
>>> cython-devel at python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> Great, thanks for the review Robert. There seems to be a refcount
>> issue with python 3:
>>
>> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
>> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
>> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
>> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> from Cython.Compiler import Main
>> [98632 refs]
>>>>> ^D
>> [98632 refs]
>> python: Modules/gcmodule.c:327: visit_decref: Assertion
>> `gc->gc.gc_refs != 0' failed.
>>
>> The object being visited is a UtilityCode object. I think the error
>> means the object is being visited more often than it's refcount value,
>> which means it's not increffed properly somewhere. I'm not sure how/if
>> my changes introduced this, has this problem been encountered recently
>> in Cython's master?
>
> Not that I'm aware of. The refnanny is supposed to catch this kind of
> thing, is it enabled?
>
> - Robert
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel

Yeah, it doesn't catch it. I'll try tracing the imports to see what it
is importing that is different from master. Thanks.

From d.s.seljebotn at astro.uio.no  Sat Oct 27 23:44:23 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Sat, 27 Oct 2012 23:44:23 +0200
Subject: [Cython] Fwd: Re: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark
 results (again, but this time more solid numbers)
In-Reply-To: <CAN8CLgkttfmgJkt2E1H5TMkkbWwPza6Mv757AuWN6cppHV3XyQ@mail.gmail.com>
References: <CAN8CLgkttfmgJkt2E1H5TMkkbWwPza6Mv757AuWN6cppHV3XyQ@mail.gmail.com>
Message-ID: <508C55B7.6020308@astro.uio.no>




-------- Original Message --------
Subject: 	Re: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results
(again, but this time more solid numbers)
Date: 	Sun, 28 Oct 2012 07:53:42 +1100
From: 	Tim Delaney <timothy.c.delaney at gmail.com>
To: 	python-dev at python.org



On 28 October 2012 07:40, Mark Shannon <mark at hotpy.org
<mailto:mark at hotpy.org>> wrote:


     I suspect that stating and loading the .pyc files is responsible for
     most of the overhead.
     PyRun starts up quite a lot faster thanks to embedding all the
     modules in the executable:
     http://www.egenix.com/__products/python/PyRun/
     <http://www.egenix.com/products/python/PyRun/>

     Freezing all the core modules into the executable should reduce
     start up time.


That suggests a test to me that the Cython guys might be interested in
(or may well have performed in the past).? How much of the stdlib could
be compiled with Cython and used during the startup process? How much of
an effect would it have on startup times and these benchmarks if
Cython-compiled extensions were used?

I'm thinking here of elimination of .pyc interpretation and execution
(stat calls would be similar, probably slightly higher).

To be clear - I'm *not* suggesting Cython become part of the required
build toolchain. But *if* the Cython-compiled extensions prove to be
significantly faster I'm thinking maybe it could become a semi-supported
option (e.g. a HOWTO with the caveat "it worked on this particular system").

Tim Delaney


-------------- next part --------------
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/d.s.seljebotn%40astro.uio.no


From markflorisson88 at gmail.com  Sun Oct 28 13:45:15 2012
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 28 Oct 2012 12:45:15 +0000
Subject: [Cython] array expressions
In-Reply-To: <CANg26EUcO0LJXT+B-hd9eS_XymMmG72bKYmZDGU6W7sT_C0DEw@mail.gmail.com>
References: <CANg26EUygnWu+=nzG=234ZuLMcYP83MXSUnoW-xWwv25vT2OnQ@mail.gmail.com>
	<5077C88E.1030205@behnel.de>
	<CANg26EWungiprtDoqwCOi8F-z0ZeM+7WWX3+qW=mQtuv+oRHXg@mail.gmail.com>
	<CADiQ+QCNmFvn3jX4Cji8y+aab39HOzqjbeP2NOiLrh1AvmGVFw@mail.gmail.com>
	<50786917.8000303@astro.uio.no>
	<CANg26EXVOKPigUBktq+THJhn1wDG5S3uwe=kw=Rv2RoFWPGbDA@mail.gmail.com>
	<507A5933.90309@behnel.de> <507A755C.30802@astro.uio.no>
	<507A7880.7030501@behnel.de>
	<CANg26EU2+co730V99F=MCfZ3PZMS36zr1TWiAVix_1j9bcBwog@mail.gmail.com>
	<507AB883.2020109@behnel.de>
	<CANg26EU72uHCn3MdYb_jFmAS3QHiDkA3+3=239pYqpxfWJCOLg@mail.gmail.com>
	<CADiQ+QAmZhBEXjEY247iZBXVuPKv=N=h+N6NeYDdvX6M8cr3sg@mail.gmail.com>
	<CANg26EXVzfJuDwXA75Gx5ZEFw7sjqdXr2sA5jb7VGoj1Sed3vg@mail.gmail.com>
	<CADiQ+QAu28O45oBQgp65EkNRnzVRhYwQwVxe8e6j_s58CCQ6Aw@mail.gmail.com>
	<CANg26EUcO0LJXT+B-hd9eS_XymMmG72bKYmZDGU6W7sT_C0DEw@mail.gmail.com>
Message-ID: <CANg26EWUAuE2o98Ym6cQLnpCcy6Qr46WycMXumc1FqtL+DSbOw@mail.gmail.com>

On 23 October 2012 10:47, mark florisson <markflorisson88 at gmail.com> wrote:
> On 23 October 2012 02:44, Robert Bradshaw <robertwb at gmail.com> wrote:
>> On Sun, Oct 21, 2012 at 5:39 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>> On 16 October 2012 18:48, Robert Bradshaw <robertwb at gmail.com> wrote:
>>>> On Sun, Oct 14, 2012 at 6:13 AM, mark florisson
>>>> <markflorisson88 at gmail.com> wrote:
>>>>> On 14 October 2012 14:05, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>>>> mark florisson, 14.10.2012 13:59:
>>>>>>> The problem with minivect as a package is that it caters to different
>>>>>>> projects, which have different requirements. Cython and minivect are
>>>>>>> quite closely coupled, and any future change, or in the future any
>>>>>>> older version may not have the functionality Cython needs, it's not
>>>>>>> exactly a stable API at this point.
>>>>>>
>>>>>> Ok, understood.
>>>>>>
>>>>>>
>>>>>>> For instance Numba needs python
>>>>>>> 2.7, whereas Cython needs to be compatible with python 2.4.
>>>>>>>
>>>>>>> Before releasing minivect I'll verify every time that it doesn't break
>>>>>>> Cython, but I currently have no real promises for backwards or forward
>>>>>>> compatibility. And that is really because not all use cases have yet
>>>>>>> been anticipated, and some really require a change, as I've already
>>>>>>> seen with Numba.
>>>>>>>
>>>>>>> We could list minivect as a dependency, which works for
>>>>>>> easy_install/pip users, but I just foresee numerous people running
>>>>>>> into problems that didn't install with pip, and I don't think an
>>>>>>> exclusion of a 300kb addition is worth any of that.
>>>>>>
>>>>>> Fine. In that case, I'm for not making minivect a separate package at all
>>>>>> but including it directly and considering it a part of Cython (and Numba
>>>>>> etc.) until there is enough of an interface to make it a reusable separate
>>>>>> package, or at least to support a separate installation and independent
>>>>>> update. Basically, if you can't update it separately, there's no use in
>>>>>> installing it separately.
>>>>>>
>>>>>> As long as we handle this so, we should take care to keep the generic parts
>>>>>> in their separate package directory and the Cython specific parts in
>>>>>> Cython, and try to keep the interface between the two as cleanly separate
>>>>>> as possible, so that we can actually reach a point where both have an
>>>>>> interface. I would guess that the need to support Numba from the same
>>>>>> source base will encourage this kind of separation anyway.
>>>>>
>>>>> Yes, definitely.
>>>>>
>>>>>> Note that this means that minivect will fall under the release schedules of
>>>>>> Cython and Numba (independently), instead of really having its own releases.
>>>>>
>>>>> It can have its own releases as well, but currently there isn't much
>>>>> point :) Minivect can be developed independent of the releases, since
>>>>> Cython and Numba need to explicitly pull in the changes. Let's make a
>>>>> habit of squashing the minivect pulls to avoid its history.
>>>>>
>>>>> I'll also wait for Dag and Robert to see if they have a (final)
>>>>> opinion before merging the subtree.
>>>>
>>>> As I mentioned on the pull request, looks good to me. Given the
>>>> (somewhat) tight coupling with the AST but the desire to use the
>>>> codebase for multiple projects, a subtree seems to make the most sense
>>>> (until/if we have some kind of a plugin system).
>>>>
>>>> - Robert
>>>> _______________________________________________
>>>> cython-devel mailing list
>>>> cython-devel at python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>
>>> Great, thanks for the review Robert. There seems to be a refcount
>>> issue with python 3:
>>>
>>> cython at sage:/jenkins/workspaces/cython-mark-build/PYVERSION/py31/build/lib.linux-x86_64-3.1-pydebug$
>>> /jenkins/workspaces/cython-mark-build/PYVERSION/py31/python/bin/python
>>> Python 3.1.5+ (default:5a6fa1b8767f, Apr 11 2012, 23:32:58)
>>> [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)] on linux2
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> from Cython.Compiler import Main
>>> [98632 refs]
>>>>>> ^D
>>> [98632 refs]
>>> python: Modules/gcmodule.c:327: visit_decref: Assertion
>>> `gc->gc.gc_refs != 0' failed.
>>>
>>> The object being visited is a UtilityCode object. I think the error
>>> means the object is being visited more often than it's refcount value,
>>> which means it's not increffed properly somewhere. I'm not sure how/if
>>> my changes introduced this, has this problem been encountered recently
>>> in Cython's master?
>>
>> Not that I'm aware of. The refnanny is supposed to catch this kind of
>> thing, is it enabled?
>>
>> - Robert
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> Yeah, it doesn't catch it. I'll try tracing the imports to see what it
> is importing that is different from master. Thanks.

I had some time to look into it, and it appeared that the bug was in
CloneNode, it just never got triggered, unless you analyse the types
of the CloneNode. That sets is_temp to True, but the node that's being
cloned may not own its reference, but temporaries are assumed to own
it.

From ndbecker2 at gmail.com  Sun Oct 28 16:44:20 2012
From: ndbecker2 at gmail.com (Neal Becker)
Date: Sun, 28 Oct 2012 11:44:20 -0400
Subject: [Cython] provision to name cython binaries
Message-ID: <k6jjsi$gar$1@ger.gmane.org>

Does cython build have a provision to name the binaries, with a suffix?  This 
would make it easier to build co-existing versions for python2 and 3.

For example:

setup.py --suffix=3

to produce

.../bin/cython3


From robertwb at gmail.com  Tue Oct 30 05:52:42 2012
From: robertwb at gmail.com (Robert Bradshaw)
Date: Mon, 29 Oct 2012 21:52:42 -0700
Subject: [Cython] provision to name cython binaries
In-Reply-To: <k6jjsi$gar$1@ger.gmane.org>
References: <k6jjsi$gar$1@ger.gmane.org>
Message-ID: <CADiQ+QAdtCWb+5cczo7LZNSXuvZ+u3bjvxOQZsPdVRjwV-v1Lg@mail.gmail.com>

No, but we'd welcome a patch.

On Sun, Oct 28, 2012 at 8:44 AM, Neal Becker <ndbecker2 at gmail.com> wrote:
> Does cython build have a provision to name the binaries, with a suffix?  This
> would make it easier to build co-existing versions for python2 and 3.
>
> For example:
>
> setup.py --suffix=3
>
> to produce
>
> .../bin/cython3
>
> _______________________________________________
> cython-devel mailing list
> cython-devel at python.org
> http://mail.python.org/mailman/listinfo/cython-devel