Question

I am using Jsoup Java HTML parser to fetch images from a particular URL. But some of the images are throwing a status 502 error code and are not saved to my machine. Here is the code snapshot i have used:-

String url = "http://www.jabong.com";
String html = Jsoup.connect(url.toString()).get().html();
Document doc = Jsoup.parse(html, url);
images = doc.select("img");

for (Element element : images) {
        String imgSrc = element.attr("abs:src");
        log.info(imgSrc);
        if (imgSrc != "") {
            saveFromUrl(imgSrc, dirPath+"/" + nameCounter + ".jpg");
            try {
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                log.error("error in sleeping");
            }
            nameCounter++;
        }
}

And the saveFromURL function looks like this:-

public static void saveFromUrl(String Url, String destinationFile) {
    try {
        URL url = new URL(Url);
        InputStream is = url.openStream();
        OutputStream os = new FileOutputStream(destinationFile);

        byte[] b = new byte[2048];
        int length;

        while ((length = is.read(b)) != -1) {
            os.write(b, 0, length);
        }

        is.close();
        os.close();
    } catch (IOException e) {
        log.error("Error in saving file from url:" + Url);
        //e.printStackTrace();
    }
}

I searched on internet about status code 502 but it says error is due to bad gateway. I don’t understand this. One of the possible things i am thinking that this error may be because of I am sending get request to images in loop. May be webserver is not able handle to this much load so denying the request to the images when previous image is not sent.So I tried to put sleep after fetching every image but no luck 🙁
Some advices please

Answer 1

Here’s a full code example that works for me…

import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.MalformedURLException;
import java.net.Proxy;
import java.net.SocketAddress;
import java.net.URL;

public class DownloadImage {

    public static void main(String[] args) {

        // URLs for Images we wish to download
        String[] urls = {
                "http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png",
                "http://www.google.co.uk/images/srpr/logo3w.png",
                "http://i.microsoft.com/global/en-us/homepage/PublishingImages/sprites/microsoft_gray.png"
                };

        for(int i = 0; i < urls.length; i++) {
            downloadFromUrl(urls[i]);
        }

    }

    /*
    Extract the file name from the URL
    */
    private static String getOutputFileName(URL url) {

        String[] urlParts = url.getPath().split("/");

        return "c:/temp/" + urlParts[urlParts.length-1];
    }

    /*
    Assumes there is no Proxy server involved.
    */
    private static void downloadFromUrl(String urlString) {

        InputStream is = null;
        FileOutputStream fos = null; 

        try {
            URL url = new URL(urlString);

            System.out.println("Reading..." + url);

            HttpURLConnection conn = (HttpURLConnection)url.openConnection(proxy);

            is = conn.getInputStream(); 

            String filename = getOutputFileName(url);

            fos = new FileOutputStream(filename);

            byte[] readData = new byte[1024];

            int i = is.read(readData);

            while(i != -1) {
                fos.write(readData, 0, i);
                i = is.read(readData);
            }

            System.out.println("Created file: " + filename);
        }
        catch (MalformedURLException e) {
            e.printStackTrace();
        }
        catch (IOException e) {
            e.printStackTrace();
        }
        finally {
            if(is != null) {
                try {
                    is.close();
                } catch (IOException e) {
                    System.out.println("Big problems if InputStream cannot be closed");
                }
            }           
            if(fos != null) {
                try {
                    fos.close();
                } catch (IOException e) {
                    System.out.println("Big problems if FileOutputSream cannot be closed");
                }
            }
        }

        System.out.println("Completed");
    }
}

You should see the following ouput on your console…

Reading...http://cdn.sstatic.net/stackoverflow/img/apple-touch-icon.png
Created file: c:/temp/apple-touch-icon.png
Completed
Reading...http://www.google.co.uk/images/srpr/logo3w.png
Created file: c:/temp/logo3w.png
Completed
Reading...http://i.microsoft.com/global/en-us/homepage/PublishingImages/sprites/microsoft_gray.png
Created file: c:/temp/microsoft_gray.png
Completed

So that’s a working example without a Proxy server involved.

Only if you require authentication with a proxy server here’s an additional Class that you’ll need based on this Oracle technote

import java.net.Authenticator;
import java.net.PasswordAuthentication;

public class ProxyAuthenticator extends Authenticator {

    private String userName, password;

    public ProxyAuthenticator(String userName, String password) {
        this.userName = userName;
        this.password = password;
    }

    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication(userName, password.toCharArray());
    }
}

And to use this new Class you would use the following code in place of the call to openConnection() shown above

...
try {
    URL url = new URL(urlString);

    System.out.println("Reading..." + url);

    Authenticator.setDefault(new ProxyAuthenticator("username", "password");

    SocketAddress addr = new InetSocketAddress("proxy.server.com", 80);
    Proxy proxy = new Proxy(Proxy.Type.HTTP, addr);

    HttpURLConnection conn = (HttpURLConnection)url.openConnection(proxy);

    ...