Question

With windows Azure Storage Client Library, CloudBlob.OpenRead() method reads only 4 mb of data. How can I read full stream using OpenRead method.

CloudBlob blob = container.GetBlobReference(filename);
Stream stream = blob.OpenRead();

I must use OpenRead method. Can’t use DownloadToFile or DownloadToStream.

EDIT :
The consumer code which is outside my scope calls

stream.CopyTo(readIntoStream);

CopyTo is an extension method.

public static int CopyTo(this Stream source, Stream destination)
        {
            byte[] buffer = new byte[BUFFER_SIZE];
            int bytesRead;
            int bytesCopied = 0;

            do
            {
                bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
                if (bytesRead > 0)
                    destination.Write(buffer, 0, bytesRead);
                bytesCopied += bytesRead;
            }
            while (bytesRead > 0);

            return bytesCopied;
        }

EDIT 2 :

I have observed that when file is uploaded using blob.UploadText() method, it works fine but when blob is uploaded using OpenWrite method as done in following code sample, the OpenRead Method reads only 4194304 bytes (4 mb).

using(var input = File.OpenRead(@"C:\testFile.txt")) //5000000 bytes
using (var output = blob.OpenWrite())
{
       input.CopyTo(output);
}

EDIT 3 :

Complete code that produces the issue at my end.

 static void Main(string[] args)
    {
        var blobContainer = GetContainer("tier1");
        blobContainer.CreateIfNotExist();

        var containerPermissions = new BlobContainerPermissions();
        containerPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;
        blobContainer.SetPermissions(containerPermissions);
        var blob = blobContainer.GetBlobReference("testfileWithOpenWrite1.txt");

        using (var input = File.OpenRead(@"C:\testFile.txt")) //5000000 bytes
        using (var output = blob.OpenWrite(new BlobRequestOptions()))
            input.CopyTo(output);

        Console.WriteLine("uploaded");

        int bytesDownloaded = 0;
        byte[] buffer = new byte[65536];

        using (BlobStream bs = blob.OpenRead())
        {
            int chunkLength;
            do
            {
                chunkLength = bs.Read(buffer, 0, buffer.Length);
                bytesDownloaded += chunkLength;
            } while (chunkLength > 0);
        }

        Console.WriteLine(bytesDownloaded);
    }

    public static int CopyTo(this Stream source, Stream destination)
    {
        int BUFFER_SIZE = 65536;
        byte[] buffer = new byte[BUFFER_SIZE];
        int bytesRead;
        int bytesCopied = 0;

        do
        {
            bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
            if (bytesRead > 0)
                destination.Write(buffer, 0, bytesRead);
            bytesCopied += bytesRead;
        }
        while (bytesRead > 0);

        return bytesCopied;
    }

EDIT 4 – Conclusion:

This was probably a bug in Microsoft.WindowsAzure.StorageClient.dll (assembly version 6.0.6002.17043) that comes with SDK v1.2. It works with latest Microsoft.WindowsAzure.StorageClient.dll (assembly version 6.0.6002.18312 – SDK v1.4).

Thanks smarx 🙂

Answer 1

I can’t reproduce what you’re seeing. Things seem to work as expected:

static void Main(string[] args)
{
    // I also tried a real cloud storage account. Same result.
    var container = CloudStorageAccount.DevelopmentStorageAccount.CreateCloudBlobClient().GetContainerReference("testcontainer");
    container.CreateIfNotExist();

    var blob = container.GetBlobReference("testblob.txt");

    blob.UploadText(new String('x', 5000000));

    var source = blob.OpenRead();

    int BUFFER_SIZE = 4000000;
    byte[] buffer = new byte[BUFFER_SIZE];
    int bytesRead;
    int bytesCopied = 0;
    do
    {
        bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
        bytesCopied += bytesRead;
    } while (bytesRead > 0);

    Console.WriteLine(bytesCopied); // prints 5000000
}

EDIT:

I’ve also (in response to the edited question) now tried uploading the blob using OpenWrite, and the result is the same. (I get the full blob back.) I used this code in place of blob.UploadText(…):

using (var input = File.OpenRead(@"testblob.txt")) //5000000 bytes
using (var output = blob.OpenWrite())
{
    input.CopyTo(output);
}

The final WriteLine still prints 5000000.

EDIT 2:

This is getting a bit tiresome. Changing the BUFFER_SIZE to 65536 didn’t change anything. The results still seem correct to me. Here’s my full application for comparison:

static void Main(string[] args)
{
    // I also tried a real cloud storage account. Same result.
    var container = CloudStorageAccount.DevelopmentStorageAccount.CreateCloudBlobClient().GetContainerReference("testcontainer");
    container.CreateIfNotExist();

    var blob = container.GetBlobReference("testblob.txt");

    using (var input = File.OpenRead(@"testblob.txt")) //5000000 bytes
    using (var output = blob.OpenWrite())
    {
        input.CopyTo(output);
    }

    var source = blob.OpenRead();

    int BUFFER_SIZE = 65536;
    byte[] buffer = new byte[BUFFER_SIZE];
    int bytesRead;
    int bytesCopied = 0;
    do
    {
        bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
        bytesCopied += bytesRead;
    } while (bytesRead > 0);

    Console.WriteLine(bytesCopied); // prints 5000000
}