Question

I have files stored in one container within a blob storage account. I need to create a zip file in a second container containing the files from the first container.

I have a solution that works using a worker role and DotNetZip but because the zip file could end up being 1GB in size I am concerned that doing all the work in-process, using MemoryStream objects etc. is not the best way of doing this. My biggest concern is that of memory usage and freeing up resources given that this process could happen several times a day.

Below is some very stripped down code showing the basic process in the worker role:

using (ZipFile zipFile = new ZipFile())
{
    foreach (var uri in uriCollection)
    {
        var blob = new CloudBlob(uri);

        byte[] fileBytes = blob.DownloadByteArray();

        using (var fileStream = new MemoryStream(fileBytes))
        {
            fileStream.Seek(0, SeekOrigin.Begin);

            byte[] bytes = CryptoHelp.EncryptAsBytes(fileStream, "password", null);

            zipFile.AddEntry("entry name", bytes);
        }
    }

    using (var zipStream = new MemoryStream())
    {
        zipFile.Save(zipStream);
        zipStream.Seek(0, SeekOrigin.Begin);

        var blobRef = ContainerDirectory.GetBlobReference("output uri");
        blobRef.UploadFromStream(zipStream);
    }

}

Can someone suggest a better approach please?

Answer 1

At the time of writing this question, I was unaware of the LocalStorage options available in Azure. I was able to write files individually to this and the work with them within the LocalStorage and then write them back to blob storage.