Question

we started a paper on Haskell a few weeks ago and just received our first assignment. I’m aware that SO doesn’t like homework questions, so I’m not going to ask how to do it. Instead, it would be very much appreciated if anyone could push me in the right direction with this. Seeing as it might not be a specific question, would it be more appropriate in a discussion / community wiki?

Question: Tokenize a String, that is: “Hello, World!” -> [“Hello”, “World”]

Coming from a Java background, I have to forget everything about the usual way to go about this. The problem is that I am still very clueless with Haskell. This is what I’ve come up with:

module Main where

main :: IO()
main = do putStrLn "Type in a string:\n"
          x <- getLine
          putStrLn "The string entered was:"
          putStrLn x
          putStrLn "\n"
          print (tokenize x)

tokenize :: String -> [String]
tokenize [] = []
tokenize l = token l ++ tokenize l

token :: String -> String
token [] = []
token l = takeWhile (isAlphaNum) l

What would be the first glaring mistake?
Thank you.

Answer 1

The first glaring mistake is

tokenize l = token l ++ tokenize l

(++) :: [a] -> [a] -> [a] appends two lists of the same type. Since token :: String -> String (and type String = [Char]), the type of tokenize that is inferred from that line is tokenize :: String -> String.
You should use (:) :: a -> [a] -> [a] here.

The next mistake in that line is that in the recursive call, you pass the same input l once again, so you have an infinite recursion, always doing the same without change. You have to remove the first token (and a bit more) from the input for the argument to the recursive call.

Another problem is that your token supposes that the input begins with alphanumeric characters.

You also need a function that ensures that condition for what you pass to token.