From cbarton at metavr.com  Tue Sep  4 02:16:27 2007
From: cbarton at metavr.com (Campbell Barton)
Date: Tue, 04 Sep 2007 10:16:27 +1000
Subject: [capi-sig] Assigning values to functions from C
In-Reply-To: <46C2C44E.9010005@behnel.de>
References: <114389462927509@lycos-europe.com> <46C2C44E.9010005@behnel.de>
Message-ID: <46DCA3DB.5080903@metavr.com>

Hi, I need to assign a value to a function (PyCFunction) from the C apy

The python equivalent is...
---
def foo(): pass
foo.Value = 10
---

Iv looked a fair bit in python source code and cant see how this is 
possible...


The PyCFunction struct looks like this...


typedef struct {
     PyObject_HEAD
     PyMethodDef *m_ml; /* Description of the C function to call */
     PyObject    *m_self; /* Passed as 'self' arg to the C func, can be 
NULL */
     PyObject    *m_module; /* The __module__ attribute, can be anything */
} PyCFunctionObject;


m_ml cant be used, m_module looks like its for internal use only, so if 
this is possible in C, its peobably stored in *m_self,

Anyone know a solution for this?

- Cam


From hniksic at xemacs.org  Tue Sep  4 09:49:53 2007
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Tue, 04 Sep 2007 09:49:53 +0200
Subject: [capi-sig] Assigning values to functions from C
In-Reply-To: <46DCA3DB.5080903@metavr.com> (Campbell Barton's message 
	of"Tue, 04 Sep 2007 10:16:27 +1000")
References: <114389462927509@lycos-europe.com> 
	<46C2C44E.9010005@behnel.de><46DCA3DB.5080903@metavr.com>
Message-ID: <874piauaxq.fsf@mulj.homelinux.net>

Campbell Barton <cbarton at metavr.com> writes:

> Iv looked a fair bit in python source code and cant see how this is
> possible...

The thing is, a Python-defined function isn't stored in a PyCFunction,
it's stored in the PyFunction object, which includes a func_dict
member.

I'm afraid you won't be able to easily assign values to a PyCFunction.
You can try to inherit PyCFunction and add a dict to it, or you can
create your own callable type that has a dict.

From hniksic at xemacs.org  Tue Sep  4 11:30:47 2007
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Tue, 04 Sep 2007 11:30:47 +0200
Subject: [capi-sig] Threading
In-Reply-To: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com> 
	(PabloYabo's message of "Mon, 13 Aug 2007 18:51:27 -0300")
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>
Message-ID: <87zm02srp4.fsf@mulj.homelinux.net>

"Pablo Yabo" <pablo.yabo at gmail.com> writes:

> The problem is that I having problems with threads. I saw that I
> have to PyEval_InitThreads and then PyThreadState_New and
> PyThreadState_Swap from the new thread but this way to solve the
> problem doesn't work for me because I need to let 2 or more threads
> run at the SAME time.

I don't know if this is still relevant, but it occurs to me that what
you want might be accomplished.  If your external library is
thread-safe, and if you're only using Python to manipulate data
managed by the library, you can simply Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS when entering the library.  That way all calls
from Python into the library will run with threading enabled, while
the part spent in the interpreter (outside the library entry points)
will continue to run single thread at a time.  Python sources often
use this pattern when dealing with IO-bound (e.g. the file and socket
API) or CPU-bound code.

Is such a solution acceptable for you?

From hniksic at xemacs.org  Tue Sep  4 14:19:28 2007
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Tue, 04 Sep 2007 14:19:28 +0200
Subject: [capi-sig] _PyObject_GC_Malloc
Message-ID: <87myw2sjvz.fsf@mulj.homelinux.net>

Does anyone know why the functionality of _PyObject_GC_Malloc isn't
available in a public function?  I need to create a GCable object
whose size is computed at instance creation time, but which is not a
classic VAR_ object (it doesn't have an ob_size field).

Currently I call _PyObject_GC_Malloc directly, but I don't like the
dependence on a private API.  If there is no good reason for keeping
gc_malloc private, I'd like to propose on python-dev that it be made
public as PyObject_GC_Malloc.

From pablo.yabo at gmail.com  Tue Sep  4 14:50:46 2007
From: pablo.yabo at gmail.com (Pablo Yabo)
Date: Tue, 4 Sep 2007 09:50:46 -0300
Subject: [capi-sig] Threading
In-Reply-To: <87zm02srp4.fsf@mulj.homelinux.net>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>
	<87zm02srp4.fsf@mulj.homelinux.net>
Message-ID: <db6698600709040550p54b2a9edjbfddb3ae99f96c5d@mail.gmail.com>

Hello Hrvoje,

Thank you for your response, it's difficult to get an answer on this
subject.

I 'solved' the issue in that way. When I enter to any function of my library
I release the interpreter lock and then I get it again.
It's not the best solution but it's acceptable. The problem is that Python
code cannot use any native wait or suspend function.

I wonder if why this issue is not fixed.

Regards,
Pablo Yabo

--
http://www.nektra.com

On 9/4/07, Hrvoje Niksic <hniksic at xemacs.org> wrote:
>
> "Pablo Yabo" <pablo.yabo at gmail.com> writes:
>
> > The problem is that I having problems with threads. I saw that I
> > have to PyEval_InitThreads and then PyThreadState_New and
> > PyThreadState_Swap from the new thread but this way to solve the
> > problem doesn't work for me because I need to let 2 or more threads
> > run at the SAME time.
>
> I don't know if this is still relevant, but it occurs to me that what
> you want might be accomplished.  If your external library is
> thread-safe, and if you're only using Python to manipulate data
> managed by the library, you can simply Py_BEGIN_ALLOW_THREADS and
> Py_END_ALLOW_THREADS when entering the library.  That way all calls
> from Python into the library will run with threading enabled, while
> the part spent in the interpreter (outside the library entry points)
> will continue to run single thread at a time.  Python sources often
> use this pattern when dealing with IO-bound (e.g. the file and socket
> API) or CPU-bound code.
>
> Is such a solution acceptable for you?
>

From hniksic at xemacs.org  Wed Sep  5 09:55:28 2007
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Wed, 05 Sep 2007 09:55:28 +0200
Subject: [capi-sig] Threading
In-Reply-To: <db6698600709040550p54b2a9edjbfddb3ae99f96c5d@mail.gmail.com> 
	(PabloYabo's message of "Tue, 4 Sep 2007 09:50:46 -0300")
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com><87
	zm02srp4.fsf@mulj.homelinux.net><db6698600709040550p54b2a9edjbfddb3ae99f96c
	5d@mail.gmail.com>
Message-ID: <87k5r535sf.fsf@mulj.homelinux.net>

"Pablo Yabo" <pablo.yabo at gmail.com> writes:

> I 'solved' the issue in that way. When I enter to any function of my
> library I release the interpreter lock and then I get it again.

OK, that is what I proposed.

> It's not the best solution but it's acceptable. The problem is that
> Python code cannot use any native wait or suspend function.

What do you mean by "native wait or suspend"?  When entering blocking
OS calls such as os.wait, or even the stdio IO code, Python tries to
release the interpreter lock, to enable other threads to run at the
same time.

> I wonder if why this issue is not fixed.

Because it is very hard to fix without slowing down the interpreter
for the single-thread case.

From pablo.yabo at gmail.com  Wed Sep  5 17:42:58 2007
From: pablo.yabo at gmail.com (Pablo Yabo)
Date: Wed, 5 Sep 2007 12:42:58 -0300
Subject: [capi-sig] Threading
In-Reply-To: <87k5r535sf.fsf@mulj.homelinux.net>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>
	<db6698600709040550p54b2a9edjbfddb3ae99f96c5d@mail.gmail.com>
	<87k5r535sf.fsf@mulj.homelinux.net>
Message-ID: <db6698600709050842u42a144afx8d2f16eb7bede93d@mail.gmail.com>

> > It's not the best solution but it's acceptable. The problem is that
> > Python code cannot use any native wait or suspend function.
>
> What do you mean by "native wait or suspend"?  When entering blocking
> OS calls such as os.wait, or even the stdio IO code, Python tries to
> release the interpreter lock, to enable other threads to run at the
> same time.


Great, I didn?t know that. I mean that if a thread waits for a condition of
another thread or sleeps but I think that they should release interpreter
lock also.

Thanks for you help.

Pablo Yabo
--
http://www.nektra.com

From hniksic at xemacs.org  Wed Sep  5 22:39:31 2007
From: hniksic at xemacs.org (Hrvoje Niksic)
Date: Wed, 05 Sep 2007 22:39:31 +0200
Subject: [capi-sig] Threading
In-Reply-To: <db6698600709050842u42a144afx8d2f16eb7bede93d@mail.gmail.com> 
	(PabloYabo's message of "Wed, 5 Sep 2007 12:42:58 -0300")
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com><db
	6698600709040550p54b2a9edjbfddb3ae99f96c5d@mail.gmail.com><87k5r535sf.fsf@m
	ulj.homelinux.net><db6698600709050842u42a144afx8d2f16eb7bede93d@mail.gmail.
	com>
Message-ID: <874pi8c0e4.fsf@mulj.homelinux.net>

"Pablo Yabo" <pablo.yabo at gmail.com> writes:

> Great, I didn?t know that. I mean that if a thread waits for a
> condition of another thread or sleeps but I think that they should
> release interpreter lock also.

And you're right.  After all, time.sleep is an easy and obvious target
for releasing the GIL, and releasing it when waiting for a condition
is necessary to avoid deadlock.

From cbarton at metavr.com  Thu Sep  6 06:51:09 2007
From: cbarton at metavr.com (Campbell Barton)
Date: Thu, 06 Sep 2007 14:51:09 +1000
Subject: [capi-sig] Best Place to start a C/API Cookbook
In-Reply-To: <874pi8c0e4.fsf@mulj.homelinux.net>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com><db	6698600709040550p54b2a9edjbfddb3ae99f96c5d@mail.gmail.com><87k5r535sf.fsf@m	ulj.homelinux.net><db6698600709050842u42a144afx8d2f16eb7bede93d@mail.gmail.	com>
	<874pi8c0e4.fsf@mulj.homelinux.net>
Message-ID: <46DF873D.50809@metavr.com>

Hi, Id like to start a C/API cookbook and collect generic useful utility 
functions there that can be a resource for people who are newer to the C 
api.

This page dosnt really ave any sections that match a C/API cookbook, and 
you need a login to do so.
http://wiki.python.org/moin/


We could start a wikibook here at http://en.wikibooks.org

A while back I started one for Blender3D/Python
http://en.wikibooks.org/wiki/Blender_3D:_Blending_Into_Python/Cookbook

There are quite a few generic functions in Blender3D's python API that 
Id add in there and Im sure others here have some code they could 
contribute too.

- Cam

-- 
Campbell J Barton (ideasman42)

From fdrake at gmail.com  Thu Sep  6 14:22:42 2007
From: fdrake at gmail.com (Fred Drake)
Date: Thu, 6 Sep 2007 08:22:42 -0400
Subject: [capi-sig] Best Place to start a C/API Cookbook
In-Reply-To: <46DF873D.50809@metavr.com>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>
	<874pi8c0e4.fsf@mulj.homelinux.net> <46DF873D.50809@metavr.com>
Message-ID: <9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>

On 9/6/07, Campbell Barton <cbarton at metavr.com> wrote:
> Hi, Id like to start a C/API cookbook and collect generic useful utility
> functions there that can be a resource for people who are newer to the C
> api.

Good idea.

> This page dosnt really ave any sections that match a C/API cookbook, and
> you need a login to do so.
> http://wiki.python.org/moin/

The Python wiki does *not* require a login.  I'd suggest that this
should be the preferred place to start such an effort unless there's a
compelling reason it won't work.  I don't see one at this point.


  -Fred

-- 
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller

From cbarton at metavr.com  Fri Sep  7 01:46:43 2007
From: cbarton at metavr.com (Campbell Barton)
Date: Fri, 07 Sep 2007 09:46:43 +1000
Subject: [capi-sig] Best Place to start a C/API Cookbook
In-Reply-To: <9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>	
	<874pi8c0e4.fsf@mulj.homelinux.net> <46DF873D.50809@metavr.com>
	<9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>
Message-ID: <46E09163.3010707@metavr.com>

Fred Drake wrote:
> On 9/6/07, Campbell Barton <cbarton at metavr.com> wrote:
>> Hi, Id like to start a C/API cookbook and collect generic useful utility
>> functions there that can be a resource for people who are newer to the C
>> api.
> 
> Good idea.
> 
>> This page dosnt really ave any sections that match a C/API cookbook, and
>> you need a login to do so.
>> http://wiki.python.org/moin/
> 
> The Python wiki does *not* require a login.  I'd suggest that this
> should be the preferred place to start such an effort unless there's a
> compelling reason it won't work.  I don't see one at this point.
> 
> 
>   -Fred

Started the cookbook here,
http://wiki.python.org/moin/CAPI_CookBook

Its linked from this page
http://wiki.python.org/moin/CategoryDocumentation

Need to come up with a prefix for helper functions like this... so far 
Iv left the prefix we had in blender.

Please add helper functions from your own API projects. ;)

From cbarton at metavr.com  Sun Sep  9 05:11:31 2007
From: cbarton at metavr.com (Campbell Barton)
Date: Sun, 09 Sep 2007 13:11:31 +1000
Subject: [capi-sig] Subtyping builtins in C
In-Reply-To: <46E09163.3010707@metavr.com>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>		<874pi8c0e4.fsf@mulj.homelinux.net>
	<46DF873D.50809@metavr.com>	<9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>
	<46E09163.3010707@metavr.com>
Message-ID: <46E36463.10401@metavr.com>

Hi, this compiles fine on linux but on windows with MSVC8 it gives an 
error on this line.

	&PyFloat_Type,		/* tp_base */
saying that it cant initialize because &PyFloat_Type is not a constant.

Has anyone else had this problem or know a workaround?

Heres teh pytype,
source file is here
http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup
http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup

PyTypeObject BPyFloat_Type = {
	PyObject_HEAD_INIT(NULL)
	0,
	"BPyFloat",
	sizeof(BPyFloatObject),
	0,
	(destructor)BPyFloat_dealloc,	/* tp_dealloc */
	0,					/* tp_print */
	0,					/* tp_getattr */
	0,					/* tp_setattr */
	0,					/* tp_compare */
	0,					/* tp_repr */
	0,					/* tp_as_number */
	0,					/* tp_as_sequence */
	0,					/* tp_as_mapping */
	0,					/* tp_hash */
	0,					/* tp_call */
	0,					/* tp_str */
	0,					/* tp_getattro */
	0,					/* tp_setattro */
	0,					/* tp_as_buffer */
	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
	0,					/* tp_doc */
	0,					/* tp_traverse */
	0,					/* tp_clear */
	0,					/* tp_richcompare */
	0,					/* tp_weaklistoffset */
	0,					/* tp_iter */
	0,					/* tp_iternext */
	BPyFloat_methods,	/* tp_methods */
	0,					/* tp_members */
	BPyFloat_getset,	/* tp_getset */
	&PyFloat_Type,		/* tp_base */
	0,					/* tp_dict */
	0,					/* tp_descr_get */
	0,					/* tp_descr_set */
	0,					/* tp_dictoffset */
	(initproc)BPyFloat_init, /* tp_init */
	0,					/* tp_alloc */
	0,					/* tp_new */
};

From cbarton at metavr.com  Sun Sep  9 05:33:32 2007
From: cbarton at metavr.com (Campbell Barton)
Date: Sun, 09 Sep 2007 13:33:32 +1000
Subject: [capi-sig] Subtyping builtins in C
In-Reply-To: <46E36463.10401@metavr.com>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>		<874pi8c0e4.fsf@mulj.homelinux.net>	<46DF873D.50809@metavr.com>	<9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>	<46E09163.3010707@metavr.com>
	<46E36463.10401@metavr.com>
Message-ID: <46E3698C.9090909@metavr.com>

found the problem from re-reading pythons source.
  they have to be zero because some compilers complain,

The values need to be assigned in the module initializer before 
PyType_Ready is called,

xxsubtype.c uses a macro DEFERRED_ADDRESS so the value is documented but 
not compiled in.


Campbell Barton wrote:
> Hi, this compiles fine on linux but on windows with MSVC8 it gives an 
> error on this line.
> 
> 	&PyFloat_Type,		/* tp_base */
> saying that it cant initialize because &PyFloat_Type is not a constant.
> 
> Has anyone else had this problem or know a workaround?
> 
> Heres teh pytype,
> source file is here
> http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup
> http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup
> 


From khughes at pacific.edu  Sun Sep  9 05:48:38 2007
From: khughes at pacific.edu (Ken Hughes)
Date: Sat, 08 Sep 2007 20:48:38 -0700
Subject: [capi-sig] Subtyping builtins in C
In-Reply-To: <46E36463.10401@metavr.com>
References: <db6698600708131451v6cd8fd34j4f317fc4fe3f393a@mail.gmail.com>		<874pi8c0e4.fsf@mulj.homelinux.net>	<46DF873D.50809@metavr.com>	<9cee7ab80709060522i7786f66bnd210e2e809160bd6@mail.gmail.com>	<46E09163.3010707@metavr.com>
	<46E36463.10401@metavr.com>
Message-ID: <46E36D16.50407@pacific.edu>

The work-around for a similar problem with tp_new is described here:

http://docs.python.org/ext/dnt-basics.html

"On some platforms or compilers, we can't statically initialize a 
structure member with a function defined in another C module, so, 
instead, we'll assign the tp_new slot in the module initialization 
function just before calling  PyType_Ready()"

So in BPyFloatType_Init(), you put the initialization for 
"BPyFloat_Type.tp_base = &PyFloat_Type;"

Ken

Campbell Barton wrote:
> Hi, this compiles fine on linux but on windows with MSVC8 it gives an 
> error on this line.
>
> 	&PyFloat_Type,		/* tp_base */
> saying that it cant initialize because &PyFloat_Type is not a constant.
>
> Has anyone else had this problem or know a workaround?
>
> Heres teh pytype,
> source file is here
> http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.c?root=bf-blender&view=markup
> http://projects.blender.org/plugins/scmsvn/viewcvs.php/branches/pyapi_devel/source/blender/python/api2_2x/bpy_float.h?root=bf-blender&view=markup
>
> PyTypeObject BPyFloat_Type = {
> 	PyObject_HEAD_INIT(NULL)
> 	0,
> 	"BPyFloat",
> 	sizeof(BPyFloatObject),
> 	0,
> 	(destructor)BPyFloat_dealloc,	/* tp_dealloc */
> 	0,					/* tp_print */
> 	0,					/* tp_getattr */
> 	0,					/* tp_setattr */
> 	0,					/* tp_compare */
> 	0,					/* tp_repr */
> 	0,					/* tp_as_number */
> 	0,					/* tp_as_sequence */
> 	0,					/* tp_as_mapping */
> 	0,					/* tp_hash */
> 	0,					/* tp_call */
> 	0,					/* tp_str */
> 	0,					/* tp_getattro */
> 	0,					/* tp_setattro */
> 	0,					/* tp_as_buffer */
> 	Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
> 	0,					/* tp_doc */
> 	0,					/* tp_traverse */
> 	0,					/* tp_clear */
> 	0,					/* tp_richcompare */
> 	0,					/* tp_weaklistoffset */
> 	0,					/* tp_iter */
> 	0,					/* tp_iternext */
> 	BPyFloat_methods,	/* tp_methods */
> 	0,					/* tp_members */
> 	BPyFloat_getset,	/* tp_getset */
> 	&PyFloat_Type,		/* tp_base */
> 	0,					/* tp_dict */
> 	0,					/* tp_descr_get */
> 	0,					/* tp_descr_set */
> 	0,					/* tp_dictoffset */
> 	(initproc)BPyFloat_init, /* tp_init */
> 	0,					/* tp_alloc */
> 	0,					/* tp_new */
> };
> _______________________________________________
> capi-sig mailing list
> capi-sig at python.org
> http://mail.python.org/mailman/listinfo/capi-sig
>
>   


From gjcarneiro at gmail.com  Mon Sep 24 18:41:49 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Mon, 24 Sep 2007 17:41:49 +0100
Subject: [capi-sig] capi-sig missing in http://www.python.org/community/sigs/
Message-ID: <a467ca4f0709240941j60635dfcw370bcbb3871bcbfa@mail.gmail.com>

Hi,

This list, capi-sig, is not listed in the list of SIGs in python.org:

 http://www.python.org/community/sigs/

Just so you know.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

From jeff at taupro.com  Tue Sep 25 06:02:49 2007
From: jeff at taupro.com (Jeff Rush)
Date: Mon, 24 Sep 2007 23:02:49 -0500
Subject: [capi-sig] capi-sig missing in
	http://www.python.org/community/sigs/
In-Reply-To: <a467ca4f0709240941j60635dfcw370bcbb3871bcbfa@mail.gmail.com>
References: <a467ca4f0709240941j60635dfcw370bcbb3871bcbfa@mail.gmail.com>
Message-ID: <46F88869.1040904@taupro.com>

Gustavo Carneiro wrote:
> 
> This list, capi-sig, is not listed in the list of SIGs in python.org:
> 
>  http://www.python.org/community/sigs/
> 
> Just so you know.

I just added it to the roster of SIGs, along with some standard boilerplate.
No list of SIG objectives have been decided upon by the group, as far as I
know so I left that part blank.  Also the SIGs section of python.org relies
upon some tricky (to me) .yml files so it remains to be seen if the new entry
will appear on the site w/o needing to be kicked by some yml-generator program.

-Jeff

From gjcarneiro at gmail.com  Tue Sep 25 11:21:14 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Tue, 25 Sep 2007 10:21:14 +0100
Subject: [capi-sig] Announcing project PyBindGen
Message-ID: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>

  Hello all,

  I'd like to announce a new project for producing python extensions, called
PyBindGen.  It follows a radical new path for Python bindings generation: it
is itself written in (gasp!) Python!  And it does not even use C++ templates
at all! :-)

  The project is young but has been slowly developed for a few months now,
so it has already reached a point where it is useful for many tasks.  For
now, it has only a Python interface.  No interface description file or
header scanning supported yet.  It also has been focused more on C++ code
for now, although it was designed to also support pure C from the start.

  More information in: https://launchpad.net/pybindgen/

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

From python_capi at behnel.de  Tue Sep 25 12:14:46 2007
From: python_capi at behnel.de (Stefan Behnel)
Date: Tue, 25 Sep 2007 12:14:46 +0200
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
Message-ID: <46F8DF96.7070706@behnel.de>


Gustavo Carneiro wrote:
>   I'd like to announce a new project for producing python extensions, called
> PyBindGen.  It follows a radical new path for Python bindings generation: it
> is itself written in (gasp!) Python!

As are the wrapper generators, as it seems:

http://codebrowse.launchpad.net/~gjc/pybindgen/devel/annotate/gjc%40inescporto.pt-20070924134815-90pequnwfrfy3i3c?file_id=foomodulegen.py-20070602135951-l3yn3vrhs0ctn6ux-1

Do you really have to write all of this by hand to 'generate' a wrapper? How
do you make it functional? (or does it require an exact API mapping?)

I can't help the feeling that Pyrex/Cython code is quite a bit more readable -
and definitely less verbose. Any chance you could motivate your design a little?

Stefan


From python_capi at behnel.de  Tue Sep 25 11:55:44 2007
From: python_capi at behnel.de (Stefan Behnel)
Date: Tue, 25 Sep 2007 11:55:44 +0200
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
Message-ID: <46F8DB20.5030102@behnel.de>


Gustavo Carneiro wrote:
> I'd like to announce a new project for producing python extensions

Launchpad lists lots of (non-existing?) releases and shows a Bazaar history of
four months. May I ask in what sense this project is "new"?

The front page also says:

"""
Generation is controlled exclusively by a Python API, no limited command-line
interface or yet another interface definition file format
"""

There doesn't seem to be any documentation. Could you give a simple example
for that?


>   More information in: https://launchpad.net/pybindgen/

You should create a PyPI page for it. People are more likely to look for it there.

Stefan

From gjcarneiro at gmail.com  Tue Sep 25 12:15:49 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Tue, 25 Sep 2007 11:15:49 +0100
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <46F8DB20.5030102@behnel.de>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
	<46F8DB20.5030102@behnel.de>
Message-ID: <a467ca4f0709250315t78fd076bo1dfe678cf779f473@mail.gmail.com>

On 25/09/2007, Stefan Behnel <python_capi at behnel.de> wrote:
>
>
> Gustavo Carneiro wrote:
> > I'd like to announce a new project for producing python extensions
>
> Launchpad lists lots of (non-existing?) releases and shows a Bazaar
> history of
> four months. May I ask in what sense this project is "new"?


It is new in the sense that it has never been officially announced :)

The front page also says:
>
> """
> Generation is controlled exclusively by a Python API, no limited
> command-line
> interface or yet another interface definition file format
> """
>
> There doesn't seem to be any documentation. Could you give a simple
> example
> for that?


http://codebrowse.launchpad.net/~gjc/pybindgen/devel/files/gjc%40inescporto.pt-20070924134815-90pequnwfrfy3i3c?file_id=examples-20070531165635-sbpc6ceu9uq4cvaa-1

(sorry for the long URL)

Basically the examples/ directory contains a sample module.  Incidentally it
evolved partially into a set of unit tests in addition to being examples.

>   More information in: https://launchpad.net/pybindgen/
>
> You should create a PyPI page for it. People are more likely to look for
> it there.


Thanks for the tip; I'll try.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

From gjcarneiro at gmail.com  Tue Sep 25 12:39:33 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Tue, 25 Sep 2007 11:39:33 +0100
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <46F8DF96.7070706@behnel.de>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
	<46F8DF96.7070706@behnel.de>
Message-ID: <a467ca4f0709250339t447ec558t175443bedf58ea0e@mail.gmail.com>

On 25/09/2007, Stefan Behnel <python_capi at behnel.de> wrote:
>
>
> Gustavo Carneiro wrote:
> >   I'd like to announce a new project for producing python extensions,
> called
> > PyBindGen.  It follows a radical new path for Python bindings
> generation: it
> > is itself written in (gasp!) Python!
>
> As are the wrapper generators, as it seems:
>
>
> http://codebrowse.launchpad.net/~gjc/pybindgen/devel/annotate/gjc%40inescporto.pt-20070924134815-90pequnwfrfy3i3c?file_id=foomodulegen.py-20070602135951-l3yn3vrhs0ctn6ux-1
>
> Do you really have to write all of this by hand to 'generate' a wrapper?
> How
> do you make it functional? (or does it require an exact API mapping?)


I don't understand what "exact API mapping" means.

But yes, so far you have to write all this by hand.  Although would love to
have automatic header file scanning, one day... unfortunately I have not had
enough time for that, and would rather focus on the code generation for
now.  Lots of neat things could be done later on top of the pybindgen python
module interface...

I can't help the feeling that Pyrex/Cython code is quite a bit more readable
> -
> and definitely less verbose. Any chance you could motivate your design a
> little?


I don't  like having to learn a new language for this.  Just like
Boost.Python users are comfortable with C++ and avoid having to learn a new
language, PyBindGen users are comfortable with Python and don't have to
learn a new language.

Besides, pyrex/cython C++ support is poor or nonexistent.  Although I
acknowledge that if you wrap pure C libraries then this point is moot.

Finally, have you ever looked at the code generated by pyrex?  It is rather
scary...

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

From python_capi at behnel.de  Tue Sep 25 13:40:08 2007
From: python_capi at behnel.de (Stefan Behnel)
Date: Tue, 25 Sep 2007 13:40:08 +0200
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <a467ca4f0709250339t447ec558t175443bedf58ea0e@mail.gmail.com>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>	
	<46F8DF96.7070706@behnel.de>
	<a467ca4f0709250339t447ec558t175443bedf58ea0e@mail.gmail.com>
Message-ID: <46F8F398.5090101@behnel.de>


Gustavo Carneiro wrote:
> On 25/09/2007, *Stefan Behnel* <python_capi at behnel.de
> <mailto:python_capi at behnel.de>> wrote:
> 
> I don't understand what "exact API mapping" means.

What I meant was: is the C(++) API mapped directly to Python or can you write
abstraction code in between?


> But yes, so far you have to write all this by hand.  Although would love
> to have automatic header file scanning, one day... unfortunately I have
> not had enough time for that, and would rather focus on the code
> generation for now.  Lots of neat things could be done later on top of
> the pybindgen python module interface...

But that's what most wrapper generators are there for: move the work from the
developer into a generator. If you have to write code for every
function/method/class you wrap, I think you're better of with Pyrex/Cython, as
Pyrex code is readable and meaningful Python, not just a wrapper writer script.


> I don't  like having to learn a new language for this.  Just like
> Boost.Python users are comfortable with C++ and avoid having to learn a
> new language, PyBindGen users are comfortable with Python and don't have
> to learn a new language.

Pyrex is *almost* Python, no new language to learn. And Cython is even closer,
as it features a couple of Python 2.5 and Python 3k features.


> Besides, pyrex/cython C++ support is poor or nonexistent.  Although I
> acknowledge that if you wrap pure C libraries then this point is moot.

There are a couple of patches that improve the support. Admittedly, Pyrex does
not target C++ libraries, so OO support is mostly missing. But nothing keeps
you from implementing an object interface against C++ classes. I bet the
Cython project would be happy to include it.


> Finally, have you ever looked at the code generated by pyrex?

Yes, definitely, it's pretty readable, well optimised (with Cython, that is)
and straight forward (ok, loops are ugly and over-optimised - but *very*
fast). And I totally like the Cython feature of keeping the surrounding Cython
code inside a C comment. That way, you immediately know what you are looking at.

Stefan

From gjcarneiro at gmail.com  Tue Sep 25 14:21:34 2007
From: gjcarneiro at gmail.com (Gustavo Carneiro)
Date: Tue, 25 Sep 2007 13:21:34 +0100
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <46F8F398.5090101@behnel.de>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>
	<46F8DF96.7070706@behnel.de>
	<a467ca4f0709250339t447ec558t175443bedf58ea0e@mail.gmail.com>
	<46F8F398.5090101@behnel.de>
Message-ID: <a467ca4f0709250521jbf386e6ga7bcaa38224f48f2@mail.gmail.com>

On 25/09/2007, Stefan Behnel <python_capi at behnel.de> wrote:
>
>
> Gustavo Carneiro wrote:
> > On 25/09/2007, *Stefan Behnel* <python_capi at behnel.de
> > <mailto:python_capi at behnel.de>> wrote:
> >
> > I don't understand what "exact API mapping" means.
>
> What I meant was: is the C(++) API mapped directly to Python or can you
> write
> abstraction code in between?


Generally it requires a direct mapping.  Why would you want to write code in
the middle?  Although recently I added support for adding "user functions"
to make them appear as methods of objects.

Anyway that's not the point.  The PyBindGen design is not set in stone...

> But yes, so far you have to write all this by hand.  Although would love
> > to have automatic header file scanning, one day... unfortunately I have
> > not had enough time for that, and would rather focus on the code
> > generation for now.  Lots of neat things could be done later on top of
> > the pybindgen python module interface...
>
> But that's what most wrapper generators are there for: move the work from
> the
> developer into a generator. If you have to write code for every
> function/method/class you wrap, I think you're better of with
> Pyrex/Cython, as
> Pyrex code is readable and meaningful Python, not just a wrapper writer
> script.


Come on, I wouldn't say this interface is unreadable.  It's stuff like this:

    mod = Module('foo')
    Foo = CppClass('Foo', automatic_type_narrowing=True)
    mod.add_class(Foo)
    Foo.add_static_attribute(ReturnValue.new('int'), 'instance_count')
    Foo.add_constructor(
        CppConstructor([Parameter.new('std::string', 'datum')]))
    Foo.add_constructor(CppConstructor([]))
    Foo.add_method(CppMethod(ReturnValue.new('std::string'), 'get_datum',
[]))

Not that hard to write this, is it?

> I don't  like having to learn a new language for this.  Just like
> > Boost.Python users are comfortable with C++ and avoid having to learn a
> > new language, PyBindGen users are comfortable with Python and don't have
> > to learn a new language.
>
> Pyrex is *almost* Python, no new language to learn. And Cython is even
> closer,
> as it features a couple of Python 2.5 and Python 3k features.
>
>
> > Besides, pyrex/cython C++ support is poor or nonexistent.  Although I
> > acknowledge that if you wrap pure C libraries then this point is moot.
>
> There are a couple of patches that improve the support. Admittedly, Pyrex
> does
> not target C++ libraries, so OO support is mostly missing. But nothing
> keeps
> you from implementing an object interface against C++ classes. I bet the
> Cython project would be happy to include it.
>
>
> > Finally, have you ever looked at the code generated by pyrex?
>
> Yes, definitely, it's pretty readable, well optimised (with Cython, that
> is)
> and straight forward (ok, loops are ugly and over-optimised - but *very*
> fast). And I totally like the Cython feature of keeping the surrounding
> Cython
> code inside a C comment. That way, you immediately know what you are
> looking at.


For instance, I don't find this very readable:

static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self,
PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/
static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self,
PyObject *__pyx_args, PyObject *__pyx_kwds) {
  PyObject *__pyx_r;
  PyObject *__pyx_1 = 0;
  static char *__pyx_argnames[] = {0};
  if (unlikely(!PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "",
__pyx_argnames))) return 0;
  Py_INCREF(__pyx_v_self);

  /* "/Users/robert/sage/pyrex/cython-0.9.6.3/Demos/spam.pyx":16
 *
 *   def get_amount(self):
 *     return self.amount             # <<<<<<<<<<<<<<
 *
 *   def set_amount(self, new_amount):
 */
  __pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam
*)__pyx_v_self)->amount); if (unlikely(!__pyx_1)) {__pyx_filename =
__pyx_f[0]; __pyx_lineno = 16; goto __pyx_L1;}
  __pyx_r = __pyx_1;
  __pyx_1 = 0;
  goto __pyx_L0;

  __pyx_r = Py_None; Py_INCREF(Py_None);
  goto __pyx_L0;
  __pyx_L1:;
  Py_XDECREF(__pyx_1);
  __Pyx_AddTraceback("spam.Spam.get_amount");
  __pyx_r = 0;
  __pyx_L0:;
  Py_DECREF(__pyx_v_self);
  return __pyx_r;
}


Compared with this code generated by pybindgen:

static PyObject *
_wrap_fooinvoke_some_object_get_prefix()
{
    PyObject *py_retval;
    std::string retval;

    retval = invoke_some_object_get_prefix();
    py_retval = Py_BuildValue("s#", retval.c_str(), retval.size());
    return py_retval;
}

Pybindgen generated code is almost as clean as the code you would write by
hand.  To me that counts a lot.  I know some people don't care what happens
underneath the tools they use as long as it works.  I am not that kind of
person.

-- 
Gustavo J. A. M. Carneiro
INESC Porto, Telecommunications and Multimedia Unit
"The universe is always one step beyond logic." -- Frank Herbert

From python_capi at behnel.de  Tue Sep 25 15:12:47 2007
From: python_capi at behnel.de (Stefan Behnel)
Date: Tue, 25 Sep 2007 15:12:47 +0200
Subject: [capi-sig] Announcing project PyBindGen
In-Reply-To: <a467ca4f0709250521jbf386e6ga7bcaa38224f48f2@mail.gmail.com>
References: <a467ca4f0709250221v4e62ea89v46efbddc1337bcaf@mail.gmail.com>	
	<46F8DF96.7070706@behnel.de>	
	<a467ca4f0709250339t447ec558t175443bedf58ea0e@mail.gmail.com>	
	<46F8F398.5090101@behnel.de>
	<a467ca4f0709250521jbf386e6ga7bcaa38224f48f2@mail.gmail.com>
Message-ID: <46F9094F.20909@behnel.de>


Gustavo Carneiro wrote:
> Come on, I wouldn't say this interface is unreadable.  It's stuff like this:
> 
>     mod = Module('foo')
>     Foo = CppClass('Foo', automatic_type_narrowing=True)
>     mod.add_class(Foo)
>     Foo.add_static_attribute(ReturnValue.new('int'), 'instance_count')
>     Foo.add_constructor(
>         CppConstructor([Parameter.new('std::string', 'datum')]))
>     Foo.add_constructor(CppConstructor([]))
>     Foo.add_method(CppMethod(ReturnValue.new('std::string'),
> 'get_datum', []))
> 
> Not that hard to write this, is it?

I didn't say it's unreadable and I didn't say it was hard to write. I'm just
saying that you shouldn't have to write that at all if all you get is a
straight mapping between a C(++) API and a Python API. That's what wrapper
generators are there for: parse the interface description and generate the
wrapper with as little human interaction as possible. And, that's what the
wrapper languages (that you don't want to require people to learn) are good at.


> For instance, I don't find this very readable:
> 
> static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self,
> PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/

So far for the signature.


> static PyObject *__pyx_f_4spam_4Spam_get_amount(PyObject *__pyx_v_self,
> PyObject *__pyx_args, PyObject *__pyx_kwds) {
>   PyObject *__pyx_r;
>   PyObject *__pyx_1 = 0;
>   static char *__pyx_argnames[] = {0};
>   if (unlikely(!PyArg_ParseTupleAndKeywords(__pyx_args, __pyx_kwds, "",
> __pyx_argnames))) return 0;
>   Py_INCREF(__pyx_v_self);

That's for argument parsing - no need to look at that.


>   /* "/Users/robert/sage/pyrex/cython-0.9.6.3/Demos/spam.pyx
> <http://0.9.6.3/Demos/spam.pyx>":16
>  *
>  *   def get_amount(self):
>  *     return self.amount             # <<<<<<<<<<<<<<
>  *
>  *   def set_amount(self, new_amount):
>  */

Here we go, this is the code that was translated into the C code below.


>   __pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam
> *)__pyx_v_self)->amount); if (unlikely(!__pyx_1)) {__pyx_filename =
> __pyx_f[0]; __pyx_lineno = 16; goto __pyx_L1;}
>   __pyx_r = __pyx_1;
>   __pyx_1 = 0;
>   goto __pyx_L0;

The first line is so long to keep you from looking at it, so if you ignore the
error handling, what you see is:

__pyx_1 = PyInt_FromLong(((struct __pyx_obj_4spam_Spam *)__pyx_v_self)->amount);

__pyx_r = __pyx_1;

Relatively straight assignment code, split into code for getting the value and
code for setting the value.


>   __pyx_r = Py_None; Py_INCREF(Py_None);
>   goto __pyx_L0;
>   __pyx_L1:;
>   Py_XDECREF(__pyx_1);
>   __Pyx_AddTraceback("spam.Spam.get_amount ");
>   __pyx_r = 0;
>   __pyx_L0:;
>   Py_DECREF(__pyx_v_self);
>   return __pyx_r;
> }

Again, ignore the clean up code.

Admittedly, it's far from hand-optimised, aesthetic code, but it's definitely
readable.


> Compared with this code generated by pybindgen:
> 
> static PyObject *
> _wrap_fooinvoke_some_object_get_prefix()
> {
>     PyObject *py_retval;
>     std::string retval;
> 
>     retval = invoke_some_object_get_prefix();
>     py_retval = Py_BuildValue("s#", retval.c_str(), retval.size());
>     return py_retval;
> }

Ok, but that's almost exactly the code you wrote by hand, step by step through
your Python API. So the question is: what's the advantage compared to writing
the code *directly* by hand, i.e. in plain C?

Oh, and: you're not passing any arguments here. If you were, your function
would look a bit more bloated, just like the Pyrex code.


> Pybindgen generated code is almost as clean as the code you would write
> by hand.  To me that counts a lot.  I know some people don't care what
> happens underneath the tools they use as long as it works.  I am not
> that kind of person.

I do care a lot, that's why I keep submitting patches to Cython. The better
the code gets that Cython generates, the less I have to care about the
generated C code when I write Cython code.

I think *that* is the essence of a code generator tool.

Stefan