Multi-argument append() is illegal

[Tim Peters]

[Guido van Rossum]
> I've noticed that there is some code out there that creates a list of
> tuples and uses code like list.append(a,b,c) to add the tuple (a,b,c)
> to the list.  According to the documentation, this is illegal:
> append() only takes a single argument, and one should write
> list.append((a,b,c)).  However, the actual append() implementation
> didn't mind, and implemented list.append(a,b,c) as
> list.append((a,b,c)).  Many people are using this even though it's
> never been documented.
>
> I am going to rectify this in Python 1.6 -- people coming from other
> languages might well expect list.append(a, b, c) to mean the same as
> list.append(a); list.append(b); list.append(c), and it's always been
> my philosophy to make ambiguous syntax illegal rather than to pick one
> interpretation randomly.

Before anyone starts <wink>, protesting this appears to be as futile as
griping about whitespace:  The Dictator Has Spoken here.  It's been broken
forever and needs to get fixed.

> ...
> You can also grep through your sources for a pattern like
> "\. *append *\(.*," -- which doesn't find every occurrence, but is
> a good starting point.

Actually, it's a terrible starting point:  the sheer quantity of "false
hits" will drive you mad.  For example, note that the one above matches your
.append()s again *after* you've fixed them(!).  I've been thru this a few
times in my own code over the years, and never dreamed up a regexp that
didn't leave me with hours of manual drudge.  You really need to account for
bracket nesting levels to avoid spurious hits on "deep" commas, and to
ignore commas in strings and trailing comments, and to span lines too to
avoid missing the

    list.append(long_expression1...
                ...ending_here,
                and_another)

cases.

So, Python to the rescue.  What appears to be a very capable checker is
attached.  It's pretty slow, but not achingly so, and you shouldn't need to
use it often.  Read the docstring at the top for cautions.  A copy of this
(or a fixed version, if this proves to have bugs) may show up in the 1.6
distribution.  BTW, there's no chance of talking me into trying to extend
this to edit your files for you, so it's a good thing you didn't ask <wink>.

parsing-is-better-than-wild-ass-guessing-ly y'rs  - tim


#! /usr/bin/env python

# Released to the public domain, by Tim Peters, 28 February 2000.

"""checkappend.py -- search for multi-argument .append() calls.

Usage:  specify one or more file or directory paths:
    checkappend [-v] file_or_dir [file_or_dir] ...

Each file_or_dir is checked for multi-argument .append() calls.  When
a directory, all .py files in the directory, and recursively in its
subdirectories, are checked.

Use -v for status msgs.  Use -vv for more status msgs.

In the absence of -v, the only output is pairs of the form

    filename(linenumber):
    line containing the suspicious append

Note that this finds multi-argument append calls regardless of whether
they're attached to list objects.  If a module defines a class with an
append method that takes more than one argument, calls to that method
will be listed.

Note that this will not find multi-argument list.append calls made via a
bound method object.  For example, this is not caught:

    somelist = []
    push = somelist.append
    push(1, 2, 3)
"""

__version__ = 1, 0, 0

import os
import sys
import string
import getopt
import tokenize

verbose = 0

def errprint(*args):
    msg = string.join(args)
    sys.stderr.write(msg)
    sys.stderr.write("\n")

def main():
    args = sys.argv[1:]
    global verbose
    try:
        opts, args = getopt.getopt(sys.argv[1:], "v")
    except getopt.error, msg:
        errprint(msg + "\n\n" + __doc__)
        return
    for opt, optarg in opts:
        if opt == '-v':
            verbose = verbose + 1
    if not args:
        errprint(__doc__)
        return
    for arg in args:
        check(arg)

def check(file):
    if os.path.isdir(file) and not os.path.islink(file):
        if verbose:
            print "%s: listing directory" % `file`
        names = os.listdir(file)
        for name in names:
            fullname = os.path.join(file, name)
            if ((os.path.isdir(fullname) and
                 not os.path.islink(fullname))
                or os.path.normcase(name[-3:]) == ".py"):
                check(fullname)
        return

    try:
        f = open(file)
    except IOError, msg:
        errprint("%s: I/O Error: %s" % (`file`, str(msg)))
        return

    if verbose > 1:
        print "checking", `file`, "..."

    ok = AppendChecker(file, f).run()
    if verbose and ok:
        print "%s: Clean bill of health." % `file`

[FIND_DOT,
 FIND_APPEND,
 FIND_LPAREN,
 FIND_COMMA,
 FIND_STMT]   = range(5)

class AppendChecker:
    def __init__(self, fname, file):
        self.fname = fname
        self.file = file
        self.state = FIND_DOT
        self.nerrors = 0

    def run(self):
        try:
            tokenize.tokenize(self.file.readline, self.tokeneater)
        except tokenize.TokenError, msg:
            errprint("%s: Token Error: %s" % (`self.fname`, str(msg)))
            self.nerrors = self.nerrors + 1
        return self.nerrors == 0

    def tokeneater(self, type, token, start, end, line,
                NEWLINE=tokenize.NEWLINE,
                JUNK=(tokenize.COMMENT, tokenize.NL),
                OP=tokenize.OP,
                NAME=tokenize.NAME):

        state = self.state

        if type in JUNK:
            pass

        elif state is FIND_DOT:
            if type is OP and token == ".":
                state = FIND_APPEND

        elif state is FIND_APPEND:
            if type is NAME and token == "append":
                self.line = line
                self.lineno = start[0]
                state = FIND_LPAREN
            else:
                state = FIND_DOT

        elif state is FIND_LPAREN:
            if type is OP and token == "(":
                self.level = 1
                state = FIND_COMMA
            else:
                state = FIND_DOT

        elif state is FIND_COMMA:
            if type is OP:
                if token in ("(", "{", "["):
                    self.level = self.level + 1
                elif token in (")", "}", "]"):
                    self.level = self.level - 1
                    if self.level == 0:
                        state = FIND_DOT
                elif token == "," and self.level == 1:
                    self.nerrors = self.nerrors + 1
                    print "%s(%d):\n%s" % (self.fname, self.lineno,
                                           self.line)
                    # don't gripe about this stmt again
                    state = FIND_STMT

        elif state is FIND_STMT:
            if type is NEWLINE:
                state = FIND_DOT

        else:
            raise SystemError("unknown internal state '%s'" % `state`)

        self.state = state

if __name__ == '__main__':
    main()

# end of file