Question

I’ve got some strings that I want to parse into a list of “chunks”. My strings look like this

"some text [[anchor]] some more text, [[another anchor]]. An isolated ["

And I expect to get back something like this

[
   TextChunk "some text ",
   Anchor "anchor",
   TextChunk " some more text, "
   Anchor "another anchor",
   TextChunk ". An isolated ["
]

I’ve managed to write a function and types that do what I need, but they seems overly ugly.
Is there a nicer way to do this?

data Token = TextChunk String | Anchor String deriving (Show)
data TokenizerMode = EatString | EatAnchor deriving (Show)

tokenize::[String] -> [Token]
tokenize xs =  
  let (_,_,tokens) = tokenize' (EatString, unlines xs, [TextChunk ""])
  in reverse tokens

tokenize' :: (TokenizerMode, String, [Token]) -> (TokenizerMode, String,[Token])
-- If we're starting an anchor, add a new anchor and switch modes
tokenize' (EatString, '[':'[':xs, tokens) = tokenize' (EatIdentifier, xs, (Identifier ""):tokens )
-- If we're ending an anchor ass a new text chunk and switch modes
tokenize' (EatAnchor, ']':']':xs, tokens) = tokenize' (EatString, xs, (TextChunk ""):tokens )
-- Otherwise if we've got stuff to consume append it
tokenize' (EatString, x:xs, (TextChunk t):tokens) = tokenize'( EatString, xs, (TextChunk (t++[x])):tokens)
tokenize' (EatAnchor, x:xs, (Identifier t):tokens) = tokenize'( EatAnchor, xs, (Identifier (t++[x])):tokens)
--If we've got nothing more to consume we're done.
tokenize' (EatString, [], tokens) = ( EatString, [], tokens)
--We'll only get here if we're given an invalid string
tokenize' xx = error ("Error parsing .. so far " ++ (show xx))

Answer 1

This should work, including lone brackets:

import Control.Applicative ((<$>), (<*), (*>))
import Text.Parsec

data Text = TextChunk String
          | Anchor String
          deriving Show

chunkChar = noneOf "[" <|> try (char '[' <* notFollowedBy (char '[')) 
chunk     = TextChunk <$> many1 chunkChar
anchor    = Anchor <$> (string "[[" *> many (noneOf "]") <* string "]]")
content   = many (chunk <|> anchor)

parseS :: String -> Either ParseError [Text]
parseS input = parse content "" input

Note the use of try to allow backtracking when the chunkChar parser matches two opening brackets. Without try, the first bracket would have been consumed at that point.