Question

How long does it take for a change to a file in Google Cloud Storage to propagate?

I’m having this very frustrating problem where I change the contents of a file and re-upload it via gsutil, but the change doesn’t show up for several hours. Is there a way to force a changed file to propagate everything immediately?

If I look at the file in the Google Cloud Storage console, it sees the new file, but then if I hit the public URL it’s an old version and in some cases, 2 versions ago.

Is there a header that I’m not setting?

EDIT:

I tried gsutil -h "Cache-Control: no-cache" cp -a public-read MyFile and it doesn’t help, but maybe the old file needs to expire before the new no-cache version takes over?

I did a curl -I on the file and get this back:

HTTP/1.1 200 OK
Server: HTTP Upload Server Built on Dec 12 2012 15:53:08 (1355356388)
Expires: Fri, 21 Dec 2012 19:58:39 GMT
Date: Fri, 21 Dec 2012 18:58:39 GMT
Last-Modified: Fri, 21 Dec 2012 18:53:41 GMT
ETag: "66d820174d6de17a278b327e4c3e9b4e"
x-goog-sequence-number: 3
x-goog-generation: 1356116021512000
x-goog-metageneration: 1
Content-Type: application/octet-stream
Content-Language: en
Accept-Ranges: bytes
Content-Length: 160
Cache-Control: public, max-age=3600, no-transform
Age: 3449

Which seems to indicate it will expire in an hour, despite the no-cache.

Answer 1

Google Cloud Storage provides strong data consistency: once a write completes, a read from anywhere in the world will get the most recent data.

However, if you enable caching (which by default is true for any publicly readable object), reads of that object can see a version of the object as old as the Cache-Control max-age specified on the object. If, for example, you uploaded the file like this:

gsutil cp -a public-read file gs://my_bucket/file

You can see that the max-age is 1 hour (3600 seconds):

gsutil ls -L gs://my_bucket/file
gs://my_bucket/file:
    Creation time:  Fri, 21 Dec 2012 19:59:57 GMT
    Cache-Control:  public, max-age=3600, no-transform
    Content-Length: 1065
    Content-Type:   text/plain
    ETag:       eb3fb83beedf1efffe5b8e32e8d6a65a
    ...

If you want to prevent a publicly readable object from being cached you could do:

gsutil setmeta -h Cache-Control:no-cache gs://my_bucket/file

Alternatively, you could set a shorter max-age on the object:

gsutil setmeta -h 'Cache-Control:public, max-age=600, no-transform'

Mike Schwartz, Google Cloud Storage team