[C++-sig] pyplusplus ui / API

[Allen Bierbaum]

On 2/14/06, Matthias Baas <baas at ira.uka.de> wrote:
> Allen Bierbaum wrote:
> >> So basically, if you do not want to export some declaration, you write
> >> decl.ignore = True. By default for all declarations from std namespace
> >> and built-in ones
> >> ignore equal to True. This will solve famous, :-(, filtering problem.
> >
> > In the interface I wrote I actually reversed this logic a little.  I
> > found it easier to think of adding a flag _expose_flag and setting
> > that flag false for all declarations by default.  This way the user
> > needs to explicitly say what they want exported.  In general I think
> > explicit is more clear then implicit so I think this works out well.
>
> I think you two are both talking about the same thing, even about the
> same default behavior (I don't care whether the internal variable will
> be called _ignore_flag or _expose_flag... :) But the decision in what
> way a user proceeds (i.e. ignore everything and then expose some stuff
> or expose everything and then ignore some stuff) should be left to the
> user, so I would add both, an ignore() and an expose() method (I saw
> that Allen's version already had both). These methods could even take an
> optional bool as argument so that you could also pass a flag similar to
> what Roman posted above (so decl.ignore(False) would be the same as
> decl.expose()).

My comment really came down to wether it is default behavior to expose
everything (as I think Roman suggested) or wether everything would be
hidden by default (as my example does).  Once this decision is made
you are right that the flag could be either expose or ignore and then
just flip the logic.

>
> > or if you want to expose all classes you could write:
> >
> > mod.Class(".*").expose()
>
> I like the possibility that an object (the MultiDeclWrapper) can refer
> to a collection of declarations which could even be spread over the
> entire declaration tree instead of just being a subtree.
>
> So am I right in saying that the general concept will be to select a
> collection of declarations (mod.Class(".*")) and then "decorate" them
> (.expose()), i.e. manipulate the way they are exposed (if at all)?

Yes.  For now I only have single level queries, but the
MultiDeclWrapper would work fine with decls from across the tree.  We
would just need to make a few new search methods that work work across
the entire tree.  For now I just focused on what I thought of as the
more common "simple" interface and left the advanced stuff for later.
:)

> > At least to me this is quite an improvement over the "spaghetti code"
> > filter because it is very direct and clear what is being exposed.
>
> Right, it's still "spaghetti" in a way that you have to define all this
> stuff somewhere, but the difference is that 1) you can keep everything
> related to a particular class together and 2) you can decide to split
> the stuff up into several functions or even modules as they don't depend
> on each other.

Yes, and in many simple cases you can just use a few regular
expressions to expose everything.

> >> Open question: If user set ignore to True on namespace and\or class
> >> does it mean that
> >> pyplusplus should not export whole namespace\class?
> >
> > The way I solved this was to make expose recursive by default so when
> > they expose a namespace it will expose everything in that namespace.
> > If they want to be more selective then they need to set the argument
> > to expose() to False to prevent recursion.  So in effect I avoided
> > making a decision on this one and let the user decide for themselves.
>
> Why would someone want to ignore a namespace but expose a class
> contained in the namespace? What effect does ignoring the namespace then
> have?
> And why could it be useful to ignore a class but expose a method? (this
> doesn't make any sense, does it?)

The reason for this is because of the recursive functionality of expose.

For example:

test_ns = extmod.Namespace("test")

test_ns.expose(recursive=True)  # Exposes *everything* in test
test_ns.expose(recursive=False)  # Just exposes the namespace itself

This is needed for the case where you want to expose a namespace and a
few classes inside it but you don't want to expose every little thing
in the entire namespace.

> > customize.  It works fairly well so far but there are a few things
> > that still need added:  caching support, (un)defines, mulit-file, and
> > multi-module.
>
> Wouldn't that be options that have to be stored somewhere else than the
> declaration tree. There could either be some "global" Option object or
> these things could just be arguments, for example:
>
> mod.createModule("test_mod", dir="test_mod_bindings", multiple_file=True)

Yes, exactly.  These are capabilities that I would need to be added to
the pypp_api.Module class that encapsulates parsing, filtering, and
writing out the bindings.

> >> Class module_t will contain declarations_tree property. User will be
> >> able to travel
> >> the tree and:
> >>     to set ignore flag to True or False ( no more confusion )
> >>     to set call policies ( no more call policies factory )
> >>     to set alias
> >>     to set documentation string
> >> and may be few other things, right now I don't have good definition
> >> for "other things"
>
> ...to add new methods (if the declaration is a class).
> ...to rename arguments or disable the generation of keyword arguments.
> ...to influence the way the declaration will appear in the output file
> (i.e. provide constructor arguments for the corresponding code creator).
> ...maybe to set a "priority" (for influencing the registration order).

Yes, these things would all be very nice to inject into the decl tree
for the code creator to understand.

> >> Now questions I can not answer:
> >>
> >> 0. Does DSL should be based on C++ or Python terminology?
> >>     For example: Matthias Baas and Allen Bierbaum talk about member
> >> and free functions as method.
> >>[...]
> >
> > I would like it based on C++ terminology.  Any place that my example
> > is not was an oversight by me and should be refined.
>
> Even in C++ I refer to member functions as methods and free functions as
> just functions. Well, that's just how I learned it and what I'm used to
> but using a more rigorous C++ terminology is fine with me... (I mean it
> makes sense anyway as we still have to deal with C++ as long as the
> bindings don't exist).
>
> >> 1. What about code creators? Code creators is very powerful concept.
> >> [...]
> >
> > This is an area where I am at a lose as to how to handle this.  If we
> > are "filtering" and making decisions based on the decl tree it seems
> > like the cleanest interface to the user would be to allow them to make
> > some of these customizations in that gccxml tree.  Unfortunately this
> > tree does not have the code generators yet since it is really used to
> > create the code generators.
>
> I haven't had a close look at the code creators yet, but couldn't they
> already be customized by attaching the arguments for the constructor in
> the nodes of the declaration tree? Or are there situations where you
> must work on a code creator instance directly?

I don't know since I don't have experience with it.  I would think
that we could attach nearly everything into the decl tree but there
may be some cases where this doesn't work.  Possible problems areas I
can think of right now are: adding headers, custom code, implicitly
convertible, and there are probably others.  Roman will need to
comment on this.

>
> >>     1.1 What is the user interface to inject code "X" under class "Y" ?
>
> If code "X" is a user defined C/C++ function then the "Class" interface
> could provide something like:
>
> MyClass.addMethod("<function_name_goes_here>")

This would be a nice usable interface.  Hopefully we can get there. :)

> Maybe this could even generate a new node in the declaration tree that
> allows all the stuff that the regular nodes also allow (assigning doc
> strings, call policies, etc.).
> Then there's the question where that user defined function is actually
> implemented. I think there are two possibilities, either the user
> provides the source code as a string (e.g. in an optional argument
> 'src') or he provides the name of a header file where the function is
> declared (which might be an optional argument 'header'). In the latter
> case, the user has to make sure that the corresponding *.cpp file is
> compiled and linked with the module.
>
> >>     1.2 How user can customize generated code. For example recent
> >> registration order
> >>           problem. The quick and dirty work around was to set
> >> use_keywords property of
> >>           function_t code creator to False.
>
> As indicated above, one possibility might be a "priority" parameter that
> indicates in which order the registration will appear (a lower number
> will appear before a higher number).
>
> Another possibility might be an API function set_order() which you can
> call on any declaration that you want to influence. Example:
>
> set_order(ClassA)
> set_order(ClassB)
> set_order(ClassC)
> ...

I tend to like the idea of a set_priority() method on all decl
wrappers a little better but I agree the concept should work.  This
isn't something I have run into much since I split my headers into
different files but I think the idea sounds like a good one for users.

> A sequence like this will guarantee that ClassA is registered first,
> then ClassB, then ClassC. Any declarations that haven't been explicitly
> set will appear after that (in the order determined by pyplusplus).
> There might even be an explicit object OutputFile that represents the
> output file(s) and that has a method like the above (but I believe this
> would already be the root of the code creator tree).
>
> The registration order could also be represented by the order in which
> the nodes are stored in the tree. This just means there should be an
> order defined on the children nodes (I suppose we already have that
> anyway). Some API function could be used to set a default order that is
> automatically determined by pyplusplus and then the user can reorder the
> tree as appropriate (maybe again by a similar method than above).
>
> > One idea I did have is that we graft on new nodes to the decl tree
> > that are custom to pyplusplus without extending gccxml.  For example
> > we could add a custom_code_t derived from declarations_t that could be
> > added into the declaration tree by the user to add code anywhere.
> > Similarly there could be a new node for implicitly_convertibles and so
> > on.  The main reason for doing this would be to allow the user to
> > write code like:
> >
> > test_ns = exmod.Namespace("test")
> > test_ns.addImplictConvertible("testPtr1", "testPtr2")
> > test_ns.addHeader("/usr/local/custom_include.h")
> > test_ns.addCode("""
> > /* custom code. */
> > """)
>
> test_ns is a node from the declaration tree, right? This raises the
> question if this tree should only represent the *contents* of the Python
> module or also the files and layout of the generated files.
> So far I'm rather undecided about that but I tend to leave it the way it
> is, i.e. the declaration tree represents the module content and the code
> creator tree the content of the generated files.

I don't like the idea of having to work on 2 trees.  It turns the
generation into a longer process with several phases:
- parse files
- modify decl tree
- create creators
- modify creator tree
- create output files

If possible I think we should just look at the decl tree as the
"source" for generating the creators.  So if we want to have a creator
for custom code be created then we need to put the "source" for it in
the decl tree.  I could be convinced otherwise, but this just seems
like it would provide the most consistent (ie. easy, understandable)
interface to the end user.

>
> >> 3. Technical problem: pygccxml is stand alone project. It is useful on
> >> its own. I don't want
> >>     to pollute it with code that is only useful for pyplusplus. I
> >> think that next solution will
> >>     work, but I don't like it:
> >>     pygccxml defines next class pygccxml.declaration.class_t
> >>
> >>     pyplusplus will create new class that derives from
> >> pygccxml.declaration.class_t class.
> >>     Lets say it will be pyplusplus.declaration_wrappers.class_t.
> >> pyplusplus will change
> >>     __class__ attribute of pygccxml.declaration.class_t instance to be
> >>     pyplusplus.declaration_wrappers.class_t. Thus user will have
> >> "original" declarations
> >>     tree, full functionality of pygccxml and easy way to find out what
> >> he can\should
> >>     change\set for specific declaration. What I don't like is the
> >> trick, that changes
> >>     __class__ attribute.
> >
> > Why derive a new class.  It seems like you could get by with just
> > injecting the flags/attributes that are needed to specify what the
> > creators should create.  Am I missing something here?
>
> I think what he meant (and what I initially also thought) is to directly
> work on the declaration tree, i.e. all the convenience methods that make
> up the public high level API would be part of the pygccxml classes. You
> could still decide if you use the high level API or if you work directly
> with the low level stuff.
> I was also already thinking about adding new methods to the pygccxml
> classes at runtime to get a high level API. But your way of having
> wrapper classes that are created when needed looks like the way to go.
> This actually separates the data (the declaration tree) from the
> interface (the DeclWrapper classes) which is a good thing in my opinion.
>   This way we can keep pygccxml as it is and add a public API on top of
> it (coincidentally, this is exactly the approach that is used in the SDK
> I'm currently wrapping). We could even try several APIs at the same time
> as they just provide different means for manipulating the declaration
> tree. And these API classes can ensure that the declaration tree is kept
> in a consistent and valid state.

Exactly.  If all the APIs do is inject new information into the
existing decls to provide additional "source" information to the
creators and the creator generators then we can try out many different
ways of getting the information in there.  We just need to define the
common decoration format and then we can build up from there.  To keep
it clean we could even put all the new information inside a common
contained class inside all the decls.

So for example we could have:

class DeclDecorator:
   __init__(self):
      self.expose = False
      self.call_policy = None
      self.rename = None
      ...

Then the wrapper classes could inject a new member of this type into
each decl and update it's properties as needed to give the creators
everything they need.

> Another thought I had is if it would also be possible/useful if the API
> would contain the original Boost.Python API as a subset so that you
> could directly call a method def() on your class that is used just as in
> C++. I don't know if such a thing is really useful or desired but it
> could provide a backdoor to specify things that the high level API does
> not provide for some reason.

That would be one powerful backdoor. :)   I like the idea assuming we
can get the regular interface to work well first.

If we are able to implement all this I may never have to customize a
generated boost.python binding again.  That would be a great goal. :)

-Allen

>
> - Matthias -
>
> _______________________________________________
> C++-sig mailing list
> C++-sig at python.org
> http://mail.python.org/mailman/listinfo/c++-sig
>