Functions Of Functions Returning Functions

[Lawrence D’Oliveiro]

The less code you have to write, the better. Less code means less
maintenance, and fewer opportunities for bugs. Here is an example of
how I was able to knock a few hundred lines off the size of a Python
module.

When I was writing my Python wrapper for HarfBuzz
<https://github.com/ldo/harfpy>, there were a lot of places where you
could define routines for HarfBuzz to make calls to, to customize the
type-shaping process in various ways.

Of course, HarfBuzz <http://behdad.github.io/harfbuzz/> is a library
written in C++, and it doesn’t know that the callbacks you pass are
actually written in Python. But ctypes
<https://docs.python.org/3/library/ctypes.html> provides an answer to
this: its CFUNCTYPE function lets you wrap Python functions so that
they become callable from C/C++ code.

When I said there were a lot of places for callbacks, I meant a lot of
places. For example, there is a HarfBuzz object called “hb_font_funcs”
<http://behdad.github.io/harfbuzz/harfbuzz-hb-font.html>, which
consists of nothing more than a container for 14 different action
callback routines. Not only that, but each one lets you pass a
separate “user data” pointer to the action callback, along with an
optional “destroy” callback which can do any necessary cleanup of this
data, which will be called when the hb_font_funcs object is disposed.

Imagine having to set all of these up by hand. Each API call to
install a callback would look something like this:

    def set_xxx_callback(self, callback_func, user_data, destroy) :
        "sets the xxx callback, along with an optional destroy callback" \
        " for the user_data. The callback_func should be declared as follows:\n" \
        "\n" \
        "    def callback_func(self, ... xxx-specific args ..., user_data)\n" \
        "\n" \
        "where self is the FontFuncs instance ... description of xxx-specific args."

        def def_wrap_xxx_callback(self, callback_func, user_data)
            # generates ctypes wrapper for caller-specified Python function.

            @HB.font_get_xxx_func_t
            def wrap_xxx_callback(... xxx-specific args ..., c_user_data) :
                ... convert xxx-specific args from ctypes representation to ...
                ... higher-level Python representation, pass to callback_func, ...
                ... along with user_data, then convert any result to ctypes ...
                ... representation and return as my result ...
            #end wrap_xxx_callback

        #begin def_wrap_xxx_callback
            return \
                wrap_xxx_callback
        #end def_wrap_xxx_callback

    #begin set_xxx_callback
        wrap_callback_func = def_wrap_xxx_callback(self, callback_func, user_data)
        if destroy != None :
            @HB.destroy_func_t
            def wrap_destroy(c_user_data) :
                destroy(user_data)
            #end wrap_destroy
        else :
            wrap_destroy = None
        #end if
        # save references to wrapper objects to prevent them prematurely
        # disappearing (common ctypes gotcha)
        self._wrap_xxx_func = wrap_callback_func
        self._wrap_xxx_destroy_func = wrap_destroy
        hb.hb_font_funcs_set_xxx_func(self._hbobj, wrap_callback_func, None, wrap_destroy)
    #end set_callback

Just think if you had to do this 14 times. And then there are a couple
of other HarfBuzz objects with their own similar collections of
callbacks as well...

Luckily, I figured out a way to cut the amount of code needed for this
by about half. It’s the recognition that the only part that is
different between all these callback-setting calls is the
“def_wrap_xxx_callback” function, together with a few different
attribute names elsewhere. So I encapsulated the common part of all
this setup into the following routine:

    def def_callback_wrapper(celf, method_name, docstring, callback_field_name, destroy_field_name, def_wrap_callback_func, hb_proc) :
        # Common routine for defining a set-callback method. These all have the same form,
        # where the caller specifies
        #  * the callback function
        #  * an additional user_data pointer (meaning is up the caller)
        #  * an optional destroy callback which is passed the user_data pointer
        #    when the containing object is destroyed.
        # The only variation is in the arguments and result type of the callback.

        def set_callback(self, callback_func, user_data, destroy) :
            # This becomes the actual set-callback method.
            wrap_callback_func = def_wrap_callback_func(self, callback_func, user_data)
            if destroy != None :
                @HB.destroy_func_t
                def wrap_destroy(c_user_data) :
                    destroy(user_data)
                #end wrap_destroy
            else :
                wrap_destroy = None
            #end if
            setattr(self, callback_field_name, wrap_callback_func)
            setattr(self, destroy_field_name, wrap_destroy)
            getattr(hb, hb_proc)(self._hbobj, wrap_callback_func, None, wrap_destroy)
        #end set_callback

    #begin def_callback_wrapper
        set_callback.__name__ = method_name
        set_callback.__doc__ = docstring
        setattr(celf, method_name, set_callback)
    #end def_callback_wrapper

Something else that saved even more code was noticing that some
callbacks came in pairs, sharing the same routine types. For example,
the font_h_extents and font_v_extents callbacks had matching types,
they were simply operating along different axes. For both of these, I
could define a common callback-wrapper definer, as follows:

    def def_wrap_get_font_extents_func(self, get_font_extents, user_data) :

        @HB.font_get_font_extents_func_t
        def wrap_get_font_extents(c_font, c_font_data, c_metrics, c_user_data) :
            metrics = get_font_extents(self, get_font_data(c_font_data), user_data)
            if metrics != None :
                c_metrics.ascender = metrics.ascender
                c_metrics.descender = metrics.descender
                c_metrics.line_gap = metrics.line_gap
            #end if
            return \
                metrics != None
        #end wrap_get_font_extents

    #begin def_wrap_get_font_extents_func
        return \
            wrap_get_font_extents
    #end def_wrap_get_font_extents_func

and the code for defining all 14 callbacks becomes as simple as

    for basename, def_func, protostr, resultstr in \
        (
            ("font_h_extents", def_wrap_get_font_extents_func, "get_font_h_extents(self, font_data, user_data)", " FontExtents or None"),
            ("font_v_extents", def_wrap_get_font_extents_func, "get_font_v_extents(self, font_data, user_data)", " FontExtents or None"),
            ... entries for remaining callbacks ...
        ) \
    :
        def_callback_wrapper \
          (
            celf = FontFuncs,
            method_name = "set_%s_func" % basename,
            docstring =
                    "sets the %(name)s_func callback, along with an optional destroy"
                    " callback for the user_data. The callback_func should be declared"
                    " as follows:\n"
                    "\n"
                    "    def %(proto)s\n"
                    "\n"
                    " where self is the FontFuncs instance and font_data was what was"
                    " passed to set_font_funcs for the Font, and return a%(result)s."
                %
                    {"name" : basename, "proto" : protostr, "result" : resultstr},
            callback_field_name = "_wrap_%s_func" % basename,
            destroy_field_name = "_wrap_%s_destroy" % basename,
            def_wrap_callback_func = def_func,
            hb_proc = "hb_font_funcs_set_%s_func" % basename,
          )
    #end for

The above loop is executed at the end of creating the basic FontFuncs
class, to fill in the methods for setting the callbacks.

This shows the power of functions as first-class objects. The concept
is older than object orientation, and is often left out of
object-oriented languages. I think Python benefits from the fact that
it had functions before it had classes.