Creating a python c-module: passing double arrays to c functions. segmentation fault. swig

[kmgrds at gmail.com]

Hello everybody,

I'm building a python module to do some heavy computation in C (for
dynamic time warp distance computation).

The module is working perfectly most of the time, which makes it so
difficult to track down the error. and I finally figured out that the
strange segmentation faults I get from time to time have something to
do with the length of the vectors I'm looking at. (So in this minimal
example I'm doing completely useless distance measures on vectors with
the same entry in all dimensions, but the errors seem to have nothing
to with the actual values in the vectors, only with the lengths.)

But somehow I haven't quite gotten the essential ideas of what one has
to do to debug this module and I hope someone can give me some advice.
The main problem is probably somewhere around passing the double
vectors to the c module or reserving and freeing memory.

so, for example
list1,list2 = [0.1 for i in  range(555)],[0.2 for i in  range(1874)]
ctimewarp(list1,list2) # see python file below
works perfectly fine
and if the vector has one more element
list1,list2 = [0.1 for i in  range(555)],[0.2 for i in  range(1875)]
ctimewarp(list1,list2)
it dies with a simple
Segmentation fault
nothing more said :-S

For very small lists again, no problem:
list1,list2 = [0.1 for i in  range(3)],[0.2 for i in  range(4)]
ctimewarp(list1,list2)
for intermediate size I get an error and more information:
list1,list2 = [0.1 for i in  range(22)],[0.2 for i in  range(99)]
ctimewarp(list1,list2)
give:

*** glibc detected *** python: free(): invalid next size (fast):
0x0804d090 ***
======= Backtrace: =========
/lib/i686/libc.so.6[0xb7bed4e6]
/lib/i686/libc.so.6(cfree+0x90)[0xb7bf1010]
/home/kim/Documents/pythonprojects/alignator/
_timewarpsimple.so[0xb7aadc0c]
/home/kim/Documents/pythonprojects/alignator/
_timewarpsimple.so[0xb7aae0db]
/usr/lib/libpython2.5.so.1.0(PyCFunction_Call+0x107)[0xb7d5d8e7]
======= Memory map: ========
08048000-08049000 r-xp 00000000 03:05 2344438    /usr/bin/python
08049000-0804a000 rwxp 00000000 03:05 2344438    /usr/bin/python
0804a000-080f3000 rwxp 0804a000 00:00 0          [heap]
....(truncated)


When looping over these lengths of vectors, e.g. as I did in the
python file below, it runs smooth for thousands of times before dying
at 938 for the first list length and 1110  for the second. i noticed
that the sum of the list lengths is 2048=2^11 and it often seems to
die around that but that's just a guess...

If anyone has a little advice or a code snippet on how to pass float
lists to modules or anything of that kind, i would appreciate it very
much!

thanks a lot in advance
cheers
kim


__________________________________

Below you can see the c file, the .i file for swig, and a little
python script producing the errors:


_______________ .c file:

/* File : timewarpsimple.c */

#include <sys/param.h>
#include <math.h>
#include <stdlib.h>

double timewarp(double x[], int lenx, double y[], int leny) {

// printf ("%d *******************************\n", lenx);
// printf ("%d *******************************\n", leny);
	double prev;
	double recx[lenx+1];
	double recy[leny+1];
	double warp[lenx+2][leny+2];
	int i,j;
	prev = 0.0;
   	for (i = 0; i < lenx; i++) {
		recx[i]=x[i]-prev;
		prev = x[i];
   	}
	recx[lenx]=1.0-prev;
	prev = 0.0;
   	for (i = 0; i < leny; i++) {
		recy[i]=y[i]-prev;
		prev = y[i];
   	}
	recy[leny]=1.0-prev;
// 	recency vectors are done

// 	let's warp

	warp[0][0]=0.0;
	for (i = 1; i < lenx+2; i++) {
		warp[i][0]=1.0;
   	}
	for (j = 1; j < leny+2; j++) {
		warp[0][j]=1.0;
   	}

	for (i = 1; i < lenx+2; i++) {
		for (j = 1; j < leny+2; j++) {
			warp[i][j]=fabs(recx[i-1]-recy[j-1]) + MIN(MIN(warp[i-1][j],warp[i]
[j-1]),warp[i-1][j-1]);
		}


	}

return warp[lenx+1][leny+1];
}



________________ .i file:

%module timewarpsimple
%include "carrays.i"
%array_functions(double, doubleArray);

%{
double timewarp(double x[], int lenx, double y[], int leny);
%}

double timewarp(double x[], int lenx, double y[], int leny);

________________ here's what I'm doing to compile:

swig -python timewarpsimple.i
gcc -c timewarpsimple.c timewarpsimple_wrap.c -I/usr/include/
python2.5/
ld -shared timewarpsimple.o  timewarpsimple_wrap.o -o
_timewarpsimple.so

which goes thru without any problem.

________________ .py file:

#!/usr/bin/python
# -*- coding: UTF-8 -*-

import timewarpsimple

def ctimewarp(list1,list2):
	"""
	takes two lists of numbers between 0 and 1 and computes a timewarp
distance
	"""
	print "timewarping"

	alen = len(list1)
	blen = len(list2)
	a = timewarpsimple.new_doubleArray(alen*4)         # Create the first
array
	# Why I have to reserve 4 times more space I don't know, but it's the
only way to make it work...
	b = timewarpsimple.new_doubleArray(alen*4)         # Create the
second array

	for i,p in enumerate(list1):
		timewarpsimple.doubleArray_setitem(a,i,p)  # Set a value. (a,i,0)
gives identical errors
	for i,p in enumerate(list2):
		timewarpsimple.doubleArray_setitem(b,i,p)  # Set a value

	warp = timewarpsimple.timewarp(a,alen,b,blen)
	print "result",warp
	timewarpsimple.delete_doubleArray(a)
	timewarpsimple.delete_doubleArray(b)
	#a,b = None,None
	return warp

########## the tests:

#list1,list2 = [0.1 for i in  range(22)],[0.2 for i in  range(99)]
#ctimewarp(list1,list2)

for x in range(888,1111):
	for y in range(888,1111):
		list1,list2 = [0.1 for i in  range(x)], [0.2 for i in  range(y)]
		print len(list1),len(list2),len(list1)+len(list2)
		ctimewarp(list1,list2)