Question

I have multiple EBS-backed EC2 instances running and I want to be able to take a snapshot of the EBS volume behind one of them, create a new EBS volume from that snapshot, and then mount that new EBS volume onto another as an additional drive. I know how to do this via the AWS web console, but I would like to automate the process by using the AWS Java API.

If I simply call the following commands one after another:

CreateSnapshotResult snapRes
    = ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
Snapshot snap = snapRes.getSnapshot();
CreateVolumeResult volRes
    = ec2.createVolume(new CreateVolumeRequest(snap.getSnapshotId(), aZone));
String newVolumeID = volRes.getVolume().getVolumeId();
AttachVolumeResult attachRes 
    = ec2.attachVolume(new AttachVolumeRequest(newVolumeID, instanceID, "xvdg"));

I get the following error:

Caught Exception: Snapshot 'snap-8e822cfd' is not 'completed'.
Reponse Status Code: 400
Error Code: IncorrectState
Request ID: 40bc6bad-43e0-49e6-a89a-0489744d24e6

To get around this, I obviously need to wait until the snapshot is completed before I create the new EBS volume from the snapshot. According to the Amazon docs, the possible values of Snapshot.getState() are “pending, completed, or error,” so I decided to check in with AWS to see if the snapshot is still in the “pending” state. I wrote the following code, but it has not worked:

CreateSnapshotResult snapRes
    = ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
 Snapshot snap = snapRes.getSnapshot();

 System.out.println("Snapshot request sent.");
 System.out.println("Waiting for snapshot to be created");

 String snapState = snap.getState();

 System.out.println("snapState is " + snapState);
 // Wait for the snapshot to be created
 while (snapState.equals("pending"))
 {
    Thread.sleep(1000);
    System.out.print(".");
    snapState = snapRes.getSnapshot().getState();
 }
 System.out.println("Done.");

When I run this, I get the following output:

Snapshot request sent.
Waiting for snapshot to be created
snapState is pending
.............................................

Where the dots continue to be printed until I kill the program. In the AWS Web Console, I can see that the snapshot has been created (it now has a green circle marking it as “completed”), but somehow my program has not gotten the message.

When I replace the while loop with a simple wait for a second (insert the line Thread.sleep(1000) after Snapshot snap = snapRes.getSnapshot(); in the first code snippet), the program will often create a new EBS volume without complaint (it then dies when I try to attach the volume to the new instance). Sometimes, however, I will get the IncorrectState error even after waiting for a second. I assume this means that there is some variance in the amount of time it takes to create a snapshot (even of the same EBS volume), and that one second is enough to account for some but not all of the possible delay times.

I could just increase the hard-coded delay to something sure to be longer than the expected time, but that approach has many faults (it waits unnecessarily for most of the times I will use it, it is still not guaranteed to be long enough, and it won’t translate well into a solution for the second step, mounting the EBS volume onto the instance).

I would really like to be able to check in with AWS at regular intervals, check to see if the state of the snapshot has changed, and then proceed once it has. What am I doing wrong and how should I fix my code to allow my program to dynamically determine when the snapshot has been fully created?

EDIT: I’ve tried using getProgress() rather than getState() as per the suggestion. My changed code looks like this:

String snapProgress = snap.getProgress();

System.out.println("snapProgress is " + snapProgress);
// Wait for the snapshot to be created
while (!snapProgress.equals("100%"))
{
    Thread.sleep(1000);
    System.out.print(".");
    snapProgress = snapRes.getSnapshot().getProgress();
}
System.out.println("Done.");

I get the same output as I did when using getState(). I think my problem is that the snapshot object that my code references is not being updated correctly. Is there a better way to refresh/update that object than simply calling its methods repeatedly? My suspicion is that I’m running up against some sort of issue with the way that the API handles requests.

Answer 1

Solved it. I think the problem was that the Snapshot.getState() call doesn’t actually make a new call to AWS, but keeps returning the state of the object at the time it was created (which would always be pending).

I fixed the problem by using the describeSnapshots() method:

String snapState = snap.getState();
System.out.println("snapState is " + snapState);

System.out.print("Waiting for snapshot to be created");
// Wait for the snapshot to be created
while (snapState.equals("pending"))
{
    Thread.sleep(500);
    System.out.print(".");
    DescribeSnapshotsResult describeSnapRes 
        = ec2.describeSnapshots(new DescribeSnapshotsRequest().withSnapshotIds(snap.getSnapshotId()));
snapState = describeSnapRes.getSnapshots().get(0).getState();
 }
 System.out.println("\nDone.");

This makes a proper call to AWS every time, and it works.