Parsing text file with #include and #define directives

[Arnaud Delobelle]

python at bdurham.com writes:

> I'm parsing a text file for a proprietary product that has the following
> 2 directives:
>
> #include <somefile>
> #define <name> <value>
>
> Defined constants are referenced via <#name#> syntax.
>
> I'm looking for a single text stream that results from processing a file
> containing these directives. Even better would be an iterator(?) type
> object that tracked file names and line numbers as it returns individual
> lines.
>
> Is there a Python parsing library to handle this type of task or am I
> better off writing my own?
>
> The effort to write one from scratch doesn't seem too difficult (minus
> recursive file and constant loops), but I wanted to avoid re-inventing
> the wheel if this type of component already exists.
>
> Thank you,
>
> Malcolm

I think it's straightforward enough to be dealt with simply.  Here is
a solution that doesn't handle errors but should work with well-formed
input and handles recursive expansions.

expand(filename) returns an iterator over expanded lines in the file,
inserting lines of included files.

import re

def expand(filename):
    defines = {}
    def define_repl(matchobj):
        return defines[matchobj.group(1)]
    define_regexp = re.compile('#(.+?)#')
    for line in open(filename):
        if line.startswith('#include '):
           recfilename = line.strip().split(None, 1)[1]
           for recline in expand(recfilename):
               yield recline
        elif line.startswith('#define '):
           _, name, value = line.strip().split(None, 2)
           defines[name] = value
        else:
            yield define_regexp.sub(define_repl, line)

It would be easy to modify it to keep track of line numbers and file
names.

HTH

-- 
Arnaud