Question

I have a project that has a non-standard file format something like:

var foo = 5
load 'filename.txt'
var bar = 6
list baz = [1, 2, 3, 4]

And I want to parse this into a data structure much like BeautifulSoup does. But this format isn’t supported by BeautifulSoup. What is the pythonic way to build a parse tree so that I can modify the values and re-write it out? In the end I would like to do something like:

data = parse_file('file.txt')
data.foo = data.foo * 2
data.write_file('file_new.txt')

Answer 1

Here is a solution using pyparsing… it works in your case. Beware that i’m not an expert therefore depending on your standards the code could be ugly… cheers

class ConfigFile (dict):
    """
    Configuration file data
    """

    def __init__ (self, filename):
        """
        Parses config file.
        """

        from pyparsing import Suppress, Word, alphas, alphanums, nums, \
            delimitedList, restOfLine, printables, ZeroOrMore, Group, \
            Combine

        equal = Suppress ("=")
        lbrack = Suppress ("[")
        rbrack = Suppress ("]")
        delim = Suppress ("'")

        string = Word (printables, excludeChars = "'")
        identifier = Word (alphas, alphanums + '_')

        integer = Word (nums).setParseAction (lambda t: int (t[0]))
        real = Combine( Word(nums) + '.' + Word(nums) ).setParseAction (lambda t: float(t[0]))
        value = real | integer

        var_kwd = Suppress ("var")        
        load_kwd = Suppress ("load")
        list_kwd = Suppress ("list")            

        var_stm = Group (var_kwd + identifier + equal + value +
                         restOfLine.suppress ()).setParseAction (
                             lambda tok: tok[0].insert(len(tok[0]), 0))

        load_stm = Group (load_kwd + delim + string + delim +
                          restOfLine.suppress ()).setParseAction (
                              lambda tok: tok[0].insert(len(tok[0]), 1))

        list_stm = Group (list_kwd + identifier + equal + lbrack +
                          Group ( delimitedList (value, ",") ) +
                          rbrack + restOfLine.suppress ()).setParseAction (
                              lambda tok: tok[0].insert(len(tok[0]), 2))


        cnf_file = ZeroOrMore (var_stm | load_stm | list_stm)

        lines = cnf_file.parseFile (filename)
        self._lines = []
        for line in lines:
            self._lines.append ((line[-1], line[0]))
            if line[-1] != 1: dict.__setitem__(self, line[0], line[1])            
        self.__initialized = True
        # after initialisation, setting attributes is the same as setting an item

    def __getattr__ (self, key):
        try:
            return dict.__getitem__ (self, key)
        except KeyError:
            return None


    def __setattr__ (self, key, value):
        """Maps attributes to values. Only if we are initialised"""

        # this test allows attributes to be set in the __init__ method
        if not self.__dict__.has_key ('_ConfigFile__initialized'):
            return dict.__setattr__(self, key, value)

        # any normal attributes are handled normally
        elif self.__dict__.has_key (key): 
            dict.__setattr__(self, key, value)

        # takes care of including new 'load' statements
        elif key == 'load':
            if not isinstance (value, str):
                raise ValueError, "Invalid data type"
            self._lines.append ((1, value))

        # this is called when setting new attributes after __init__
        else:
            if not isinstance (value, int) and \
                not isinstance (value, float) and \
                not isinstance (value, list):
                raise ValueError, "Invalid data type"

            if dict.has_key (self, key):
                if type(dict.__getitem__(self, key)) != type (value):
                    raise ValueError, "Cannot modify data type."
            elif not isinstance (value, list): self._lines.append ((0, key))
            else: self._lines.append ((2, key))            
            dict.__setitem__(self, key, value)


    def Write (self, filename):
        """
        Write config file.
        """
        fid = open (filename, 'w')
        for d in self._lines:
            if d[0] == 0: fid.write ("var %s = %s\n" % (d[1], str(dict.__getitem__(self, d[1]))))
            elif d[0] == 1: fid.write ("file '%s'\n" % (d[1]))
            else: fid.write ("list %s = %s\n" % (d[1], str(dict.__getitem__(self, d[1]))))


if __name__ == "__main__":

    input="""var foo = 5
load 'filename.txt'
var bar = 6
list baz = [1, 2, 3, 4]"""

    file ("test.txt", 'w').write (input)
    config = ConfigFile ("test.txt")
    # Modify existent items
    config.foo = config.foo * 2
    # Add new items
    config.foo2 = [4,5,6,7]
    config.foo3 = 12.3456
    config.load = 'filenameX.txt'
    config.load = 'filenameXX.txt'
    config.Write ("test_new.txt")

EDIT

I have modified the class to use

__getitem__, __setitem__

methods to mimic the ‘access to member’ syntax to parsed items as required by our poster. Enjoy!

PS

Overloading of the

__setitem__

method should be done with care to avoid interferences between setting of ‘normal’ attributes (class members) and the parsed items (that are accesses like attributes). The code is now fixed to avoid these problems. See the following reference http://code.activestate.com/recipes/389916/ for more info. It was funny to discover this!