Question

all

i’ve found a piece of code to parse simple ARFF file and i wanna change it to fit sparse ARFF whose data looks like:

@data
{0 12,4 37,8 First,20 'Some Thing'} 
{0 12,13 First,28 'Some Thing'}

here is the code:

def ParseFromSimpleARFF(data):
    arffFormat = Forward()
    E = CaselessLiteral("E")
    comment = '%' + restOfLine
    relationToken = Keyword('@RELATION', caseless=True)
    dataToken = Keyword('@DATA', caseless=True)
    attribToken = Keyword('@ATTRIBUTE', caseless=True)
    ident = Word( alphas, alphanums + '_-' ).setName('identifier')
    relation = Suppress(relationToken) \
               + ident.setResultsName('relation')    
    classDomain = Suppress('{') \
                  + Group(delimitedList(ident.setResultsName('domain'))).setResultsName('domains') + Suppress('}')
    attribute = Group(Suppress(attribToken)
                + Word(alphas).setResultsName('attrname')+(Word(alphas)|classDomain).setResultsName('type')).setResultsName('attribute')            
    arithSign = Word("+-",exact=1)
    realNum = Combine( Optional(arithSign) 
              + (Word( nums ) + "." + Optional( Word(nums) )|( "." + Word(nums) )) 
              + Optional( E + Optional(arithSign) + Word(nums) ))

    **#dataList = Group(delimitedList(realNum|ident)).setResultsName('record')
    dataList = Suppress('{') + Group( delimitedList(realNum|ident)).setResultsName('record')  + Suppress('}')**
    arffFormat << ( relation
                   + OneOrMore(attribute).setResultsName('attributes')
                   + dataToken
                   + OneOrMore(dataList).setResultsName('records')).setResultsName('arffdata')

    simpleARFF = arffFormat
    simpleARFF.ignore(comment)
    tokens =  simpleARFF.parseString(data) 
    return tokens

but it doesn’t work

i think i’ve to tell the program to identify the wihtespace, but i don’t know how

thanks so much

Answer 1

No, no, no! A big part of pyparsing is that “whitespace happens”! Unless you are doing something tricky with indentation-based parsing, or line-oriented data, leave the whitespace out of the parser definition.

Your issue is that what you are defining as a dataList doesn’t match the lists you are giving it. Each comma-delimited item in the list is a pair of values, which seem to permit numbers or quoted strings. So define dataList as:

dataCell = realNum|ident|quotedString
dataList = Suppress('{') + Group( delimitedList(Group(dataCell + dataCell)))  + Suppress('}')

Some other bits:

• it is not necessary to forward declare arffFormat as a Forward(). This is only necessary if data will be nested within recursive structures, that is data containing subdata. Your example doesn’t have that, just define arffFormat at the end with arffFormat = (...etc.

• x.setResultsName('name') has been replaced with simply x('name'), really cleans up the parser code

• You define a realNum (which requires a decimal point), but have only integers in your sample. I’ve shifted away from building up realNum-type expressions, in favor of using a localized regex: realNum = Regex(r"[+-]?\d+(\.\d*)?([Ee][+-]?\d+)?") will give you an expression that will accept an integer or real number. This will also allow you to delete other distracting elements, like arithSign.

You may also be going overboard on the results names. I think this will give you a pretty good insight into your data, and give a pretty simple-to-navigate structure at the end:

arffFormat = ( relation
               + OneOrMore(attribute)('attributes')
               + dataToken
               + OneOrMore(dataList)('records'))('arffdata')