Why functional Python matters

[Dave Benjamin]

It has been my increasing concern that Python is headed in a decidedly
object-oriented direction, ignoring the benefits it currently offers to
functional programmers. Before you respond, "but Python *is*
object-oriented!", please allow me to explain. I don't know how many people
I speak for, but I feel a need to speak up and hopefully influence this
direction.

When I first discovered Python, it was with "mailman", which to this day is
the best mailing list software I have ever used. Before I was even a
competent PHP programmer, I knew of one high-quality web application written
in Python. Thus, in the back of my mind, I was interested in the language
for web-releated work (which is more or less what I do). However, at the
time, nobody I knew was writing web applications in Python. They were using
either Perl, PHP, ASP, or JSP, and being a rebellious GNU/Linux lover
frustrated with my awkwardness with Perl, I naturally chose PHP.

PHP turned out to be a very fine language for the tasks I had assigned it
to. It was fast, database-friendly, and provided massive libraries of
functionality catered toward typical web needs. As I became more advanced
with PHP, I began to uncover the beauty of its array functions. To clarify
some terminology here, an "array" in PHP is like a "dictionary" in Python
except that its keys are ordered, and it is sometimes treated like a "list".
One of the nicest things about built-in data structures like these is that
they can be dumped to the console or browser very easily. The presence of
fast, built-in collection types combined with a set of useful utility
functions and an easy method to observe their state at any point have made
them invaluable to my programming projects so far.

I did not do a lot of object-oriented programming in PHP, mainly because it
didn't buy me much, and techniques that I later discovered to be more
functional in nature were more helpful. For example, I tried looking at EJB
and modeling database records with dynamically-generated classes like Entity
Beans, but once I compared this with what I had been used to using
(typically lists of dictionaries representing tables of rows), I saw few
benefits for this method of modeling data. Representing result sets as lists
of dictionaries allowed me to generalize my algorithms and streamline the
data path from the database to the HTML template. Representing rows as
objects made my code bigger, more awkward, and harder to debug.

I should also mention that PHP's support for objects is not very robust, and
I admit in advance that this undoubtedly has biased me away from their use
in general. However, I will also note that I have a CS degree from a college
that promoted object-oriented programming very heavily, and was in the midst
of settling on Java as a standard. All feelings about Java set aside, I
think I have a fair amount of experience with--and well-rounded
understanding of--the object-oriented methodology.

My interest in functional programming exploded when I discovered the
following library of functional tools for PHP by Ian Kjos:

http://functional-php.sourceforge.net/

You might say that this library is to PHP what the Xoltar toolkit is to
Python, although the Xoltar toolkit is far more advanced. Nonetheless, this
source code and accompanying site opened my eyes to the functional style of
programming, and I saw immediate benefits which I was able to apply to my
everyday work. Learning about the power of higher-order functions and the
elimination of side-effects helped me write more concise, more reliable
software than ever before. This was a huge epiphany for me.

The next thing I read will probably not surprise you if you have ever done a
Google search on functional programming. It was David Mertz's "Charming
Python" series on IBM's DeveloperWorks. His papers were not easy to digest
in one sitting, but I made a great effort to understand them and apply the
techniques he described. At this point, I should mention that I was still
programming only in PHP!

By using Ian's functional.php library, I was satisfied to find that I could
follow many of David's examples, and fill in the gaps as I went along. Those
of you who balk at Python's lambda syntax might be less harsh if you knew
the kind of syntax we have to put up with PHP:

Python: x = lambda y, z: y + z
PHP:    $x = lam('$y, $z', '$y + $z');

The more I read of David's series, the less I could effectively translate to
PHP, mostly because it ventures into constructs like list comprehensions and
generators that PHP did not support. Eventually, Python's clean, simple
syntax and straightforward approach won me over, and I began migrating my
knowledge to the mod_python platform where I could free myself from all of
those dollar signs and semicolons and play with some really powerful
functional tools. As long as I'm naming names, let me just say that Gregory
Trubetskoy is my hero for what he did with mod_python and the publisher
module. I am greatly indebted to his work.

Now, I write web applications in PHP, mod_python, Zope, and even Java. Of
all of these platforms, my preference is mod_python because it lets me
program in Python--real Python, with no restrictions, no extension modules,
no Jython-stuck-at-Python-2.1, but Python proper. I absolutely adore Python
as a web language.

In fact, as I was originally learning Python, I began to consider it the
Perfect Language(tm). I know that this is a silly thing to do, but I had
learned over a dozen languages previously, and I honestly couldn't find a
single thing wrong with Python, and instead found myself trying to recreate
Python in every other language I used (often with positive results). Now,
I'd say I'm a lot more experienced with Python, and understand its strengths
and weaknesses better, but I still see Python as approaching perfection for
reasonable values of "perfect". =)

Guido's recent musings in the newsgroups and on his presentation on "Python
Regrets" have made me a bit uneasy. In particular, they brought to light the
fact that he generally prefers the object-oriented style to the functional
style and would like to "deprecate" certain features:

 - apply, to be replaced with *-notation
  - map and filter, to be replaced with list comprehensions
   - lambda
   
   Lambda has had the thorniest history of all of these, I think. Lambda has
its roots in LISP and the Lambda calculus, and can be used to create
"anonymous functions" at runtime that can be passed around as needed. This
is a very powerful and useful feature for functional programming, as well as
for event-driven programming. It baffles me that anyone would want to remove
it.

The first problem the "lambda" keyword had was that it couldn't be nested
easily or bound to a particular local variable namespace. This resulted in a
(minor) abuse of keyword arguments, and much criticism from misguided
functional advocates for its lack of "real closures"--I heavily debated this
because a) classes and closures are strikingly similar, and b) the
keyword-argument trick wasn't really that bad, once you got used to it. In
any case, with dynamic scoping, this problem has completely disappeared.

The second "problem" that everyone seems to keep bringing up is that lambda
limits you to expressions, not statements. This makes me laugh every time,
because this is one of the things that I *loved* about Python when I was
first learning it! If you get used to writing functions that only return
expressions, you are on the road to understanding the functional style! One
thing that I recommend that anyone try as a learning experience is this: try
writing a program where every single function starts with "return". Better
yet, write your whole program with lambdas. <screams from the back row> Of
course, I don't recommend doing this sort of thing "out in industry", but I
guarantee that you will learn to write code that is more reusable,
side-effect free, and finely decomposed once you have gained some practice
writing code this way.

I would not like to see lambda support "statements". I think that would ruin
a good thing. I would not like it removed from the language. I want it the
way it is. The only thing between "lambda" and perfection is the name. It
should be "fun" or something. If I had a lambda key on my keyboard, I would
press it. <grin>

"apply" is less of a big deal to me, but I'd still like to keep it because
it allows for greater portability. I feel so more strongly about map and
filter. Yes, I know about list comprehensions. I think they are wonderful, I
thank Guido for adding them, I thank Haskell for inspiring them, and I don't
want them to ever go away. But just because you can do a similar set of
things with list comprehensions than with map and filter doesn't mean that
they are completely redundant. In fact, I believe that it is advantageous to
design functions specifically for use with map and filter. If you design
this way, you will benefit from cleaner syntax than list comprehensions can
offer. For example:

 - map(process_record, records)
 
compared with:
 
 - [process_record(record) for record in records]
  
  I'm aware that adding lambdas, nesting, multiple filters, etc. can make
map/filter much less readable than list comprehensions, but functional
decomposition is another solution. If you have to nest that much, perhaps
your functions are too complicated and need to be broken up. I'm not saying
that this is always the case, but for the sake of clarity - in certain
situations - I greatly prefer map/filter to list comprehensions. For
starters, they eliminate the need to name the temporary variable.

Rather than remove these important functional built-ins, I would like to see
two last built-ins be added: unzip (to reverse the behavior of zip), and
curry (see the example by Alex Martelli, et. al. in the Python Cookbook,
sec. 15.7). Anything else can be thrown into a "functional" standard module.
The global namespace doesn't need any more clutter, and I sympathize with
those who wish to reduce it, but--please!--not at the expense of the
functions that form the foundation of functional programming!

I have observed Guido making several comments about Python being
object-oriented from the core, and that the natural way of expressing
solutions is object-oriented. I think that Paul Prescod feels even more
strongly about this. It is true that many common functional techniques could
be re-implemented elegantly through the use of object-oriented techniques,
and the opposite is also true. You can program o-o without side effects, and
you can emulate classes with closures. However, please do not draw the
conclusion that supporting both methodologies would be redundant. 

Please resist the temptation to oversimplify the language and force users
who enjoy Python's functional-friendly features to go back to the drawing
board, especially when Python's ability to do FP was a prime reason for
people like me to migrate to Python in the first place! I don't think that
objects are the way of the future. I think they are part of the future, but
I think that they focus too much on state, where I tend to concentrate on
data flow and functionality. If Python evolves to be a language that is
increasingly hostile toward functional programming, I will (*very
reluctantly*) have to look at other languages to focus my primary work on).

Finally, I'd like to make one comment about Perl. I think that this is
important and relevant. There is clearly a difference in philosophy between
Perl culture and Python culture. Perl says There's More Than One Way To Do
It, and Python begs to differ. Perl is all-inclusive, Python is more
exclusive. However, let's not get carried away with the exclusivity! There
are clearly multiple of ways of doing many things in Python, and that's not
a bad thing. Sometimes you want a list, other times you want an iterator.
Sometimes you want a list comprehension, other times you want map/filter.
Closures? Objects! Dispatches? Polymorphism! Inheritance? Composition? Just
because Perl says you can do anything any-which-way doesn't mean that to
compete Python should force you to a single method of problem-solving.

Perl is a beautiful thing. It encourages more creativity in the sense of
language design than any other language I have ever used, including Python.
There are many things to learn from Perl, and open-mindedness is one of
them. Java, on the other hand, is extremely rigid about its way of doing
things--trying to do functional programming in Java requires the use of
bulky wrapper classes instead of true, lightweight higher-order functions,
and its lack of parametric polymorphism makes the entire type system a pain
in the butt.

Python is a nice happy medium between regularity and freedom. I'd like to
see it stay that way.

Thanks for listening,
Dave