Question

there is a very time-consuming operation which generates a dataset in my package. I would like to save this dataset and let the package rebuild it only when I manually delete the cached file. Here is my approach as part of the package:

myDataset = Module[{fname, data}, 
    fname = "cached-data.mx";
    If[FileExistsQ[fname], 
        Get[fname],
        data = Evaluate[timeConsumingOperation[]];
        Put[data, fname];
        data]
];

timeConsumingOperation[]:=Module[{},
    (* lot of work here *)
    {"data"}
];

However, instead of writing the long data set to the file, the Put command only writes one line: “timeConsumingOperation[]”, even if I wrap it with Evaluate as above. (To be true, this behaviour is not consistent, sometimes the dataset is written, sometimes not.)

How do you cache your data?

Answer 1

In the past, whenever I’ve had trouble with things evaluating it is usually when I have not correctly matched the pattern required by the function. For instance,

f[x_Integers]:= x

which won’t match anything. Instead, I meant

f[x_Integer]:=x

In your case, though, you have no pattern to match: timeConsumingOperation[].

You’re problem is more likely related to when timeConsumingOperation is defined relative to myDataset. In the code you’ve posted above, timeConsumingOperation is defined after myDataset. So, on the first run (or immediately after you’ve cleared the global variables) you would get exactly the result you’re describing because timeConsumingOperation is not defined when the code for myDataset is run.

Now, SetDelayed (:=) automatically causes the variable to be recalculated whenever it is used, and since you do not require any parameters to be passed, the square brackets are not necessary. The important point here is that timeConsumingOperation can be declared, as written, prior to myDataset because SetDelayed will cause it not to be executed until it is used.

All told, your caching methodology looks exactly how I would go about it.