Question

For a dataset which takes a very long time to open (it is a stored procedure), I would like to implement some kind of caching on the Datasnap server.

So if this dataset is loaded the first time and transferred to the client (TClientDataSet), it should not be closed and reopened for the following requests unless the server restarts or a “reload” procedure on the server is called.

So after the first open, every new client would only receive a copy (clone) of the dataset without refreshing / reloading the server side dataset.

For a simple implementation of this dataset ‘cache’ the Datasnap server datamodule must not be created per session (because for each new session, the server side dataset would be closed until the client sends the request to open the DatasetProvider). Maybe I can find a solution to clone the dataset also for session datamodules but my basic question is:

Is there a way to override methods in the DatasetProvider so that the client can still open, but not close the server-side dataset?

Answer 1

Few years ago, some DataSnap server I worked on had to pull data from a very-very slow SQL Server 7 server. I then worked out a server cache “toy” based on TClientDataSets where “cached providers” are connected to those “server ClientDataSets” which in turns reads data from file cache or from database.

Cache was refreshed based on a set of specific hard-coded rules for each dataset. When the cache needs to be refreshed, server-ClientDataSet use a provider to pull data from database via ADOQuery and then the data is saved to the app-server disk using TClientDataSet’s binary format. (it enables cache sharing between server instances).

To prevent different instances to pull information from database at the same time when it determines is time to update cache, very basic synchronization method was developed. A “control file” is created on disk during the data-retrieveral operation and deleted upon completion or failure. Before pull-data operation starts, the server instance checks for file existence. If it exists, enter a wait-loop until the file is not present and check for valid data in the .cds asociated file… and act according to that.
If file don’t exists, tries to create it, covering the very same millisecond case.

This was not a 24×7 application, just a kind of 12×6 :D. The method proved to be very good, I can’t remember a single failure for this rude synchronization during almost 3 years I was maintaining that code.. but you maybe want to create a more robust mechanism.

When there’s no need to refresh the cache, data is just loaded from disk.

All the cache work was done using the provider methods.

So, the relation was something like this:

//
//     Client                    Server
//---------------    -----------------------------------------------------------------
//                                               Cache refresh?
//
//                                                   Yes
//                                                  ----- Provider --- ADOQuery - DB
// ClientDataSet ---- Provider --- ClientDataSet --|
//                                                  ----- LoadFromFile
//                                                   No
//
//

Pseudo-code for need-to-update check and OpenDataSet was like this:

function CacheRequiresRefresh: Boolean
begin
  if not IsPresentLocalData then
    Result := True
  else if ControlRecordIsMoreRecent then
    Result := True
  else if SomeOtherCondition then
    Result := True
  else
    Result := False;
end;

function OpenDataSet;
begin
  repeat
    if CacheRequiresRefresh then
    begin
      if not ControlFilePresent then
        if CreateControlFile then
        begin
          ConnectCDSToProvider;
          CDS.Open;
        end
      else
        if ControlFilePresent then
          WaitUntilControlFileIsNotPresent
    end
    else
      CDS.LoadFromFile('filename.cds');
  until CDS.Active;
end;

I don’t have access to the code anymore, sure I can’t remember every detail, happily current servers are very good and fast enough not to need to think on this now… hope explained mechanism the way it worked. If you need clarification or further help, please commment.