I have a little script to read in, parse and derive some kind of interesting (not really) statistics from an apache log file. So far I’ve made two simple options, the total number of bytes sent in all requests in the log file, and a top 10 of the most common IP adresses.
The first “mode” is just a simple sum of all the parsed bytes. The second one is a fold over a map (Data.Map), using insertWith (+) 1' to count the occurrences.
The first one runs as I expected, most of the time spent parsing, in constant space.
42,359,709,344 bytes allocated in
the heap
72,405,840 bytes copied during GC
113,712 bytes maximum residency (1553 sample(s))
145,872 bytes maximum slop
2 MB total memory in use (0 MB lost due to fragmentation)Generation 0: 76311 collections,
0 parallel, 0.89s, 0.99s elapsed
Generation 1: 1553 collections, 0
parallel, 0.21s, 0.22s elapsedINIT time 0.00s ( 0.00s
elapsed) MUT time 21.76s (
24.82s elapsed) GC time 1.10s ( 1.20s elapsed) EXIT time
0.00s ( 0.00s elapsed) Total time 22.87s ( 26.02s elapsed)%GC time 4.8% (4.6% elapsed)
Alloc rate 1,946,258,962 bytes
per MUT secondProductivity 95.2% of total user,
83.6% of total elapsed
However, the second one does not!
49,398,834,152 bytes allocated in
the heap
580,579,208 bytes copied during GC
718,385,088 bytes maximum residency (15 sample(s))
134,532,128 bytes maximum slop
1393 MB total memory in use (172 MB lost due to fragmentation)Generation 0: 91275 collections,
0 parallel, 252.65s, 254.46s elapsed
Generation 1: 15 collections, 0
parallel, 0.12s, 0.12s elapsedINIT time 0.00s ( 0.00s
elapsed) MUT time 41.11s (
48.87s elapsed) GC time 252.77s (254.58s elapsed) EXIT time
0.00s ( 0.01s elapsed) Total time 293.88s (303.45s elapsed)%GC time 86.0% (83.9% elapsed)
Alloc rate 1,201,635,385 bytes
per MUT secondProductivity 14.0% of total user,
13.5% of total elapsed
And here is the code.
{-# LANGUAGE OverloadedStrings #-}
module Main where
import qualified Data.Attoparsec.Lazy as AL
import Data.Attoparsec.Char8 hiding (space, take)
import qualified Data.ByteString.Char8 as S
import qualified Data.ByteString.Lazy.Char8 as L
import Control.Monad (liftM)
import System.Environment (getArgs)
import Prelude hiding (takeWhile)
import qualified Data.Map as M
import Data.List (foldl', sortBy)
import Text.Printf (printf)
import Data.Maybe (fromMaybe)
type Command = String
data LogLine = LogLine {
getIP :: S.ByteString,
getIdent :: S.ByteString,
getUser :: S.ByteString,
getDate :: S.ByteString,
getReq :: S.ByteString,
getStatus :: S.ByteString,
getBytes :: S.ByteString,
getPath :: S.ByteString,
getUA :: S.ByteString
} deriving (Ord, Show, Eq)
quote, lbrack, rbrack, space :: Parser Char
quote = satisfy (== '\"')
lbrack = satisfy (== '[')
rbrack = satisfy (== ']')
space = satisfy (== ' ')
quotedVal :: Parser S.ByteString
quotedVal = do
quote
res <- takeTill (== '\"')
quote
return res
bracketedVal :: Parser S.ByteString
bracketedVal = do
lbrack
res <- takeTill (== ']')
rbrack
return res
val :: Parser S.ByteString
val = takeTill (== ' ')
line :: Parser LogLine
l ine = do
ip <- val
space
identity <- val
space
user <- val
space
date <- bracketedVal
space
req <- quotedVal
space
status <- val
space
bytes <- val
(path,ua) <- option ("","") combined
return $ LogLine ip identity user date req status bytes path ua
combined :: Parser (S.ByteString,S.ByteString)
combined = do
space
path <- quotedVal
space
ua <- quotedVal
return (path,ua)
countBytes :: [L.ByteString] -> Int
countBytes = foldl' count 0
where
count acc l = case AL.maybeResult $ AL.parse line l of
Just x -> (acc +) . maybe 0 fst . S.readInt . getBytes $ x
Nothing -> acc
countIPs :: [L.ByteString] -> M.Map S.ByteString Int
countIPs = foldl' count M.empty
where
count acc l = case AL.maybeResult $ AL.parse line l of
Just x -> M.insertWith' (+) (getIP x) 1 acc
Nothing -> acc
---------------------------------------------------------------------------------
main :: IO ()
main = do
[cmd,path] <- getArgs
dispatch cmd path
pretty :: Show a => Int -> (a, Int) -> String
pretty i (bs, n) = printf "%d: %s, %d" i (show bs) n
dispatch :: Command -> FilePath -> IO ()
dispatch cmd path = action path
where
action = fromMaybe err (lookup cmd actions)
err = printf "Error: %s is not a valid command." cmd
actions :: [(Command, FilePath -> IO ())]
actions = [("bytes", countTotalBytes)
,("ips", topListIP)]
countTotalBytes :: FilePath -> IO ()
countTotalBytes path = print . countBytes . L.lines =<< L.readFile path
topListIP :: FilePath -> IO ()
topListIP path = do
f <- liftM L.lines $ L.readFile path
let mostPopular (_,a) (_,b) = compare b a
m = countIPs f
mapM_ putStrLn . zipWith pretty [1..] . take 10 . sortBy mostPopular . M.toList $ m
Edit:
Adding +RTS -A16M reduced GC to 20%. Memory use of course unchanged.
I suggest making the following changes to the code:
I made the fields of
LogLinestrict to avoid them containing thunks referring to expressions related to parsing. It’s good practice to make fields strict, unless you really need them to be lazy.I made sure that the parse result is created as soon as possible (that’s the
$!part of the change), also to avoid delaying the parsing until you actually inspect the individual fields ofLogLine.Finally I switched to a better data structure,
HashMapfrom the unordered-containers package. Note that all functions inData.HashMap.Strictare value strict, which means we can use the plaininsertWithvariant.Note that taking a sub-string of a
ByteStringforces the original string to be retained in memory, due to sharing the underlying storage (this is the same as for Java’sString). If you want to make sure that no extra memory is retained, use thecopyfunction from thebytestringpackage. You can try to callcopyon the result of(getIP x)and see if that makes any difference. The trade-off here is using some extra computation to copy the string in return for lower space usage.Note that using
-A<high number>tend to improve performance of short running programs (i.e. benchmarks) but not necessarily on real programs. Same goes for-H. At least a higher-Hvalue (e.g. 1G) doesn’t hurt the performance of your program.