Question

I download an image from a url to an ec2 instance(wget), process it with imagemagick (convert -actions $savedfilename) and then save it to s3 (with a php api).

1. Is it possible to process the image on ec2 and save it directly to s3 without writing to the s3 volume?
2. If this is possible, does it make sense – would it be faster / cheaper?
3. Is there anything else that can be done to improve the efficiency of the process?
4. Is there an API to save to s3 directly from the shell?

Answer 1

I guess you meant “…without writing to the EBS volume” Am I right? You can pipe Wgets output directly to ImageMagicks convert, which looks like this:

wget -O - 'http://d24w6bsrhbeh9d.cloudfront.net/photo/4498158_700b_v1.jpg' | convert - test.png

Take a look at s3cmd, it will allow you to interact with S3 directly from the command line. Our example workflow will then look like this:

wget -O - 'http://d24w6bsrhbeh9d.cloudfront.net/photo/4498158_700b_v1.jpg' | convert - test.png && s3cmd put --acl-public --guess-mime-type test.png s3://example.com/images/test.png

This will give you this result, which you can filter using regex to get the public URL:

File 'test.png' stored as s3://example.com/images/test.png (xxxx bytes)
Public URL of the object is: http://example.com.s3.amazonaws.com/images/test.png

Get URL from text:

<?php
  $reg_exUrl = "/(http|https|ftp|ftps)\:\/\/[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,3}(\/\S*)?/";
  $cmd_output = "File 'test.png' stored as s3://example.com/images/test.png (xxxx bytes) Public URL of the object is: http://example.com.s3.amazonaws.com/images/test.png";
  if(preg_match($reg_exUrl, $cmd_output, $url)) {
    $image_url = $url[0];
  }
  else {
    // no url found …
  }
?>

I guess that’s an elegant way of doing your process 🙂 I’m not sure if it will be any faster or cheaper … Maybe a bit because of EBS’ bad disk I/O.