[Numpy-discussion] Re: ANN: numarray-1.2.3 -- segfault in in my C program

[Sebastian Haase]

Todd,
Thanks for working on this with me.
I did not use -fPIC and/or -shared. What is -fPIC anywhy ?
I have been using my code + makefile since numarray 0.3.6 (maybe 0.4).
I took my gcc/g++ arguments from examples in the wxWindows/wxPython.

Trying your modified(attached) c-version I get the exact same behaviour that I 
had before IF I DON'T use -shared. (I.e. works with numarray1.1; new numarray 
gives "Can't import module 'numarray.libnumarray'")  (I took -pthread out)

With -shared I get a segfault before main() - gdb:
Program received signal SIGSEGV, Segmentation fault.
0x00000001 in ?? ()

How can this be ? How can this change from numarray 1.1 to 1.2 (or 1.2.3) ?

Thanks,
Sebastian


On Tuesday 10 May 2005 11:39, Todd Miller wrote:
> Hi Sebastian,
>
> I tried building and running this but I can't even get to main().  I
> even tried the attached,  which I think is completely decoupled from
> numarray and there is the same problem.
>
> BTW, did you leave out -fPIC and -shared on the command line?
>
> To simplify things,  you might try:
>
> a. removing:
> >    PyObject *pymodMain = PyImport_ImportModule("__main__");
> >   PyObject* globals   = PyModule_GetDict(pymodMain);
> >   PyEval_InitThreads();
>
> b. get rid of threads for now
>
> c. downshift from C++ to C
>
> Bottom line:  I think this is a more fundamental "embedding" setup problem.
>
> Regards,
> Todd
>
> On Tue, 2005-05-10 at 13:31, Sebastian Haase wrote:
> > Todd,
> > I would be very thankful if you could compile and run this program:
> > <testnumarrayembedded.cpp>
> > #include <Python.h>
> > #include <numarray/libnumarray.h>
> > int main() {
> >   Py_Initialize();
> >   PyObject *pymodMain = PyImport_ImportModule("__main__");
> >   PyObject* globals   = PyModule_GetDict(pymodMain);
> >   PyEval_InitThreads();
> >
> > //	import_libnumarray();
> >   PyObject *module = PyImport_ImportModule("numarray.libnumarray");
> >   if (!module)
> > 	Py_FatalError("Can't import module 'numarray.libnumarray'");
> >   if (module != NULL) {
> > 	PyObject *module_dict = PyModule_GetDict(module);
> > 	PyObject *c_api_object =
> > 	  PyDict_GetItemString(module_dict, "_C_API");
> > 	if (PyCObject_Check(c_api_object)) {
> > 	  libnumarray_API = (void **)PyCObject_AsVoidPtr(c_api_object);
> > 	} else {
> > 	  Py_FatalError("Can't get API for module 'numarray.libnumarray'");
> > 	}
> >   }
> >   float *colMapsRGB = new float[4 * 3 * 256];
> >   int shape[3]= { 4, 3, 256 };
> >   PyObject *obj = PyBuffer_FromReadWriteMemory(colMapsRGB, 4*3 *256 *
> > sizeof(float));
> >   PyObject *obj2 =(PyObject*) NA_NewAllFromBuffer( 3, shape, tFloat32,
> > obj, 0, 0, NA_ByteOrder(), 1, 1/*writable*/) ;
> >   return 0;
> > }
> > </testnumarrayembedded.cpp>
> >
> > I compile with:
> > g++-2.95 -o testnumarrayembedded -Wall -finline-functions -g
> > -I/usr/include/ python2.2 -I/jws30/haase/myCVS/numarray/Include
> > testnumarrayembedded.cpp -pthread -lpython2.2
> >
> >
> > It runs through with setting PYTHONPATH to my old numarray(1.1)
> > directory. And now with numarray CVS it now doesn't even find the
> > library... $ ./testnumarrayembedded
> > Fatal Python error: Can't import module 'numarray.libnumarray'
> > Aborted
> >
> > Still, my non-embedded python shell runs all fine - your CVS-update
> > (-j...) suggestion didn't seem to make a difference.
> >
> > Thanks,
> > Sebastian
> >
> > On Tuesday 10 May 2005 05:13, Todd Miller wrote:
> > > On Mon, 2005-05-09 at 17:58 -0700, Sebastian Haase wrote:
> > > > Hi,
> > > >
> > > > Now I'm back to this problem. In the meantime I upgraded again:
> > > > >>> na.__version__
> > > >
> > > > '1.4.0'
> > > > All my python2.2 + numarray + my-extensions work. (Debian Woody)
> > > > Only my C++ program segfaults. It looked to me like a problem with
> > > > NA_NewAllFromBuffer -- was there any recent change ?
> > >
> > > Yes there was.  You can undo it in your private CVS checkout like this:
> > >
> > > % cd numarray/Src
> > > % cvs update -j 1.3 -1.2 libnumarray.ch
> > > % cd ..
> > > % python setup.py install --genapi --selftest
> > >
> > > > (I saw that I was using a bytestride of 0 - but it used to work and
> > > > giving a 1 for that argument still segfaults  -- is 0 OK ??)
> > >
> > > I don't know for sure.  It seems like it should be... just leads to 0
> > > strides.
> > >
> > > > Now I noticed in the gdb traceback that I chokes is the function
> > > > deferred_libnumarray_init
> > > > ((
> > > > ((#0  0x406d68d5 in PyObject_GetAttrString () from
> > > > ((/usr/lib/libpython2.2.so.0.0 #1  0x410f905e in
> > > > ((deferred_libnumarray_init () at Src/libnumarraymodule.c:149
> > > >
> > > > Is this function the standard function that always get called to init
> > > > numarray (in embedded Python AND using "normal" python) ?
> > >
> > > Yes and no.  deferred_libnumarray_init() is really for stuff that can't
> > > be done at libnumarray.so init time.  This is because of circular
> > > dependencies.
> > >
> > > The "real" numarray init function is either "import numarray" in Python
> > > or "import_array()" in C.  After that,  certain calls into the C-API
> > > trigger the deferred initialization.  deferred_libnumarray_init() is
> > > never called directly by application code.
> > >
> > > > Have you tested this with Python2.2 ?
> > >
> > > Yes, using the numarray unit tests.  NewAllFromBuffer is also used
> > > directly by the numarray ufuncs,  so it is tested to a degree by
> > > numarray itself.  My gut feel is you're hitting an embedding problem,
> > > but my gut is only so accurate.
> > >
> > > > (See for more info  below == I did follow Todd's suggestions 1 & 2,
> > > > but hope not having to recompile python from source.)
> > >
> > > That's really not hard.   A 5-10 minute exercise usually.
> > >
> > > Regards,
> > > Todd