Question

I am expecting users to upload a CSV file of max size 1MB to a web form that should fit a given format similar to:

"<String>","<String>",<Int>,<Float>

That will be processed later. I would like to verify the file fits a specified format so that the program that shall later use the file doesnt receive unexpected input and that there are no security concerns (say some injection attack against the parsing script that does some calculations and db insert).

(1) What would be the best way to go about doing this that would be fast and thorough? From what I’ve researched I could go the path of regex or something more like this. I’ve looked at the python csv module but that doesnt appear to have any built in verification.

(2) Assuming I go for a regex, can anyone direct me to towards the best way to do this? Do I match for illegal characters and reject on that? (eg. no ‘/’ ‘\’ ‘<‘ ‘>’ ‘{‘ ‘}’ etc.) or match on all legal eg. [a-zA-Z0-9]{1,10} for the string component? I’m not too familiar with regular expressions so pointers or examples would be appreciated.

EDIT:
Strings should contain no commas or quotes it would just contain a name (ie. first name, last name). And yes I forgot to add they would be double quoted.

EDIT #2:
Thanks for all the answers. Cutplace is quite interesting but is a standalone. Decided to go with pyparsing in the end because it gives more flexibility should I add more formats.

Answer 1

Pyparsing will process this data, and will be tolerant of unexpected things like spaces before and after commas, commas within quotes, etc. (csv module is too, but regex solutions force you to add “\s*” bits all over the place).

from pyparsing import *

integer = Regex(r"-?\d+").setName("integer")
integer.setParseAction(lambda tokens: int(tokens[0]))
floatnum = Regex(r"-?\d+\.\d*").setName("float")
floatnum.setParseAction(lambda tokens: float(tokens[0]))
dblQuotedString.setParseAction(removeQuotes)
COMMA = Suppress(',')
validLine = dblQuotedString + COMMA + dblQuotedString + COMMA + \
        integer + COMMA + floatnum + LineEnd()

tests = """\
"good data","good2",100,3.14
"good data" , "good2", 100, 3.14
bad, "good","good2",100,3.14
"bad","good2",100,3
"bad","good2",100.5,3
""".splitlines()

for t in tests:
    print t
    try:
        print validLine.parseString(t).asList()
    except ParseException, pe:
        print pe.markInputline('?')
        print pe.msg
    print

Prints

"good data","good2",100,3.14
['good data', 'good2', 100, 3.1400000000000001]

"good data" , "good2", 100, 3.14
['good data', 'good2', 100, 3.1400000000000001]

bad, "good","good2",100,3.14
?bad, "good","good2",100,3.14
Expected string enclosed in double quotes

"bad","good2",100,3
"bad","good2",100,?3
Expected float

"bad","good2",100.5,3
"bad","good2",100?.5,3
Expected ","

You will probably be stripping those quotation marks off at some future time, pyparsing can do that at parse time by adding:

dblQuotedString.setParseAction(removeQuotes)

If you want to add comment support to your input file, say a ‘#’ followed by the rest of the line, you can do this:

comment = '#' + restOfline
validLine.ignore(comment)

You can also add names to these fields, so that you can access them by name instead of index position (which I find gives more robust code in light of changes down the road):

validLine = dblQuotedString("key") + COMMA + dblQuotedString("title") + COMMA + \
        integer("qty") + COMMA + floatnum("price") + LineEnd()

And your post-processing code can then do this:

data = validLine.parseString(t)
print "%(key)s: %(title)s, %(qty)d in stock at $%(price).2f" % data
print data.qty*data.price