I have a need to retrieve files in my web application that are greater than 64KB in size. Right now, in v1beta1 of the JSON API, Google is only allowing uploads/downloads of 64KB in size via their JSON API. I’ve figured out how to upload files over 64KB in size using a “resumable” upload (not via the interface v1beta1 provides in its JSON API, but manually).
What I can’t figure out is a good way to download. Right now, I make the ACL public for the object I want to download, download the file, and then remove the public ACL on the object. This is not only inefficient, it is not very clean. Is there a better method I could use or am I stuck until Google provides a better means in a future version of their API?
Background Information
I am writing a GAE application and I know of the google.appengine.api.files interface. Unfortunately this does not work on live buckets while using the local dev environment, and for testing purposes, my team and I need a way to test development locally (too cumbersome to deploy to GAE among other limiting/security factors). We can interact with all other APIs except for Cloud Storage, so I’m writing a class that will use either the JSON API or AppEngine’s files interface when reading/writing/deleting from Cloud Storage. I got a working implementation, but I’m unhappy with the way I retrieve files.
Clarification from comment below:
We are downloading large amounts of information, massaging it, and storing it into Cloud Storage for consumption into BigQuery. We need to use live buckets from the dev environment because if we don’t, BigQuery won’t be able to consume data we want to test. No need to serve these files, just process them
Solution from comment on accepted Answer below:
I was able to reuse my authenticated httplib2 object from my code that interacts with the JSON API to do an authenticated GET request against the https://{bucket_name}.storage.googleapis.com/{object_name} URL endpoint, adding only the Content-Length: 0 and x-goog-api-version: 2 headers.
For downloads, the XML API is the simplest way to do an authenticated download. There are multiple ways to do authenticated downloads, as detailed on this page: https://developers.google.com/storage/docs/authentication
For a simple fetch operation, the name “XML API” could be considered misleading — you don’t need to produce any XML. If you experience an error, it will be represented as XML, but in the simplest case of a successful file download, no XML is actually involved.