Question

I’d like to write a tests for a Parsec parser. Here is the example of parser and data structure:

data Event = Event { keyEvent :: String }
    deriving Show

parseKey :: Parser Event
parseKey = do
             char '<'
             k <- many1 (letter <|> digit <|> oneOf "_")
             char '>'
             return $ Event k

I know that I need to check the parse = parse . pretty . parse property. But how should I generate the correct and incorrect test cases? Generally, how should I generate test cases for given BNF? I know about instance Arbitrary, but not much help from this.

I would appreciate if you provide a well-commented example of generator for this simple parser.

Answer 1

Testing parsers is not entirely trivial. (Depending on the complexity of the thing, as with all testing.)

One thing you can do is write an Arbitrary instance which constructs all valid expressions (or whatever it is you’re trying to parse), and then check that if you pretty-print the thing and then parse the resulting string, you get back exactly what you started with.

There are a couple of problems with that:

• If the answer is wrong, what’s broken? The parser or the printer?

• If the thing you’re parsing is complicated enough to have optional brackets and things, you need to check it works both with and without the optional brackets. Your pretty printer will only do it one way or the other, usually.

• This doesn’t check that garbage input really does get rejected (and not parsed as something strange). E.g., I’ve written plenty of Parsec parsers which will silently ignore a syntax error if it occurs at the end of the input.

In general, the only really thorough way I know of testing a parser is to just write lots and lots of manual tests by hand. (And every time you find something that parses wrong, add another test case for it.) This is essentially what GHC does with their test suit, for example.

Of course, it depends how complex your parser is, and how much assurance you want… If you’re just parsing JSON, you can probably test it fairly easily. If you’re parsing something like Markdown… my God have mercy on your soul!