Pyrex list/array

[John Machin]

On 4/06/2006 7:59 PM, Jim Lewis wrote:
> Thanks for your comments.
> 
>> You probably didn't expect the Inquisition...
> 
> Correct ;-)

Nobody does :-)

> 
>> The question might be better asked on the Pyrex mailing list.
> 
> I did not find it - where is it?

Evidently somewhere near the Hall of the Mountain King. A reference to 
it is cunningly concealed in the last place one would think of finding 
it: under the heading "Mailing List" on the Pyrex home page :-) Here: 
http://lists.copyleft.no/mailman/listinfo/pyrex ... get in quick before 
the pirate moves it again.

> 
>> Almost any Python code is also valid Pyrex code. For a start, just compile your function with Pyrex and compare the speed.
> 
> It's slower.
> 
>> What you do next is going to depend very much on what operations you are performing on  the list and the objects it contains.
> 
> Simple list of ints. Comparing sections of lists between each other.

Do you mean alist[x:x+n] == alist[y:y+n] ?
If so, that's creating two new lists each of size n, and then comparing 
those two lists. I doubt that psyco would recognize that it didn't need 
to copy the two slices. The first step might be to write functions to 
compare without copying, e.g.:

def py_slices_cmp_eq(py_list, start1, start2, size):
     """Return 1 if py_list[start1+size] == py_list[start2+size]
     else 0"""
     offset = start2 - start1
     for i in xrange(start1, start1+size):
         if py_list[i] != py_list[i+offset]:
             return 0
     return 1

See what psyco makes of that.
Then turn that into a cdef function for Pyrex.

If that's still not fast enough, then you might be in for some harder work:

Allocate memory for a C array, unpack your list into it, write 
comparison functions c_slices_cmp_* that operate on your array of ints.
There should be no Python stuff in there, only C constructs. You can 
even use memcmp() for the cmp_eq function.

Which brings us back to your original question "Do I need to convert to 
array or something so pyrex can generate tight code?" ...
1. Some of the above may help you to determine whether you need to.
2. Without any knowledge of the size of your list or what you are doing, 
we can't help you much more on whether you need to.
3. AFAICT, Pyrex doesn't do much in the way of optimisation, leaving 
that up to the C compiler. Generating tight code would depend more on 
you replacing appropriately-chosen Pythonisms with C-isms.

As for *how* to make your C array, something like this:

cdef extern from "Python.h":
    void   PyMem_Free(void *p)
    void*  PyMem_Malloc(int n) except NULL
# untested -- compiles OK :-)
cdef int * int_array_from_list(object ilist):
     cdef int n, i
     cdef int * arrayp
     n = len(ilist)
     arrayp = <int *>PyMem_Malloc(n * sizeof(int))
     for i from 0 <= i < n:
         arrayp[i] = ilist[i]
     return &arrayp[0]

Hoping some of this helps,
John