Question

I have function a f which returns a large Integer. Nearing end the program has to add up all the returned values of f. The physical memory of this computer is too limited to store all the returned values of f. So I would need to put this into a file buffer. Would TVars be able to handle Integers? Is there a solution in which I can throw all the returned values of f? Also, could the separate threads be able to read it and buffer it at the the same time?

Answer 1

Your question isn’t all that clear. From what I understand you need to store all the results that a function f has returned during the run of a program and since there are large number of such results you would like to store those results in a file. Since it would be inefficient to actually store each result as soon as it was computed you would like to implement a form of buffering.

If this is the case, you could use something like Chan for example which is an unbounded blocking FIFO queue. And to answer one of your concerns, this structure was specifically designed for concurrent access from multiple threads.

So you could run your main program where you would call f, and for each call you would also insert the result into the Chan. You would also spawn another thread which will continuously read from the Chan and write the results in a file.

Now, if the rate at which the main thread (the one that calls f) is much higher than the rate at which the other thread stores the results on disk then you are back to your original problem in which results pile up in the Chan and you remain out of memory at some point. For this particular case you could use something like BoundedChan which is similar to Chan but will block at insertion when the channel is full. In this scenario, the main thread may have to wait sometimes for the writing thread to store the results on disk, but you have the guarantee that you’ll never fill-up the memory with the many results of f.

We can actually build a nice abstraction for this. We can imagine a functional traceable that given a function f and a way to store values gives us a function which returns the same results as f but as a side effect it also stores the results for later analysis.

traceable :: (a -> b) -> (b -> IO ()) -> (a -> IO b)
traceable f store = \x -> do
    let result = f x
    store result
    return result

In your case, the program could look something like this:

f :: Int -> Int
f = ... -- implementation of f here

main = do
    ch <- newChan
    traceableF = traceable f (writeChan ch)
    forkIO $ resultWriter ch
    -- the main program which calls traceableF here ...

resultWriter :: Chan Int -> IO ()
resultWriter ch = do
    f <- obtainFileHandler
    forever $ do
        result <- readChan
        writeToFile f result

You may also need to write some logic so that the main thread waits for the resultWriter thread to finish writing to disk, but basically that’s about it.

Hope this answers your question.