Question

I like to write a process in Worker role, to download (sync) batch of files under a folder(directory) to local mirrored folder(directory)

Is there a timestamp(or a way to get) on the time of last folder(directory) updated?

Since folder(directory) structure unsure, but simply put is download whatever there to local, as soon as it changes. Except recursion and setup a timer to check it repeatedly, whats another smart idea do you have?

(edit) p.s. I found many solutions on sync files from local to Azure storage, but the same principle on local files cannot apply on Azure blob, I am still looking for a way that most easily to download(sync) files to local as soon as they are changed.

Answer 1

Eric, I believe the concept you’re trying to implement isn’t really that effective for your core requirement, if I understand it correctly.

Consider the following scenario:

1. Keep your views in the blob storage.
2. Implement Azure (AppFabric) Cache.
3. Store any view file to the cache, if it’s not yet there on a web request with unlimited(or a very long) expiration time.
4. Enable local cache on your web role instances with a short expiration time (e.g. 5 minutes)
5. Create a (single, separated) worker role, outside your web roles, which scans your blobs’ ETags for changes in interval. Reset the view’s cache key for any blob changed
6. Get rid of those ugly “workers” inside of your web roles 🙂

There’re a few things to think about in this scenario:

• Your updated views will get to the web role instances within “local cache expiration time + worker scan interval”. The lower the values, the more distributed cache requests and blob storage transactions.
• The Azure AppFabric Cache is the only Azure service preventing the whole platform to be truly scalable. You have to choose the best cache plan based on the overall size (in MB) of your views, the number of your instances and the number of simultaneous cache requests required per instance.
• consider caching of the compiled views inside your instances (not in the AppFabric cache). Reset this local cache based on the dedicated AppFabric cache key/keys. This will raise the performance greatly for you, as rendering the output html will be as easy as injecting the model to the pre-compiled views.
• of course, the cache-retrieval code in your web roles must be able to retrieve the view from the primary source (storage), if it is unable to retrieve it from the cache for whatever reason.