Question

I have to call a rest web service that returns a large amount of data as xml. The data is about 490m in size. Every time I try to call the service I run out of memory. All I want to do is write this data to a file.

Is there a way to read and write the data in small chunks to avoid running out of memory?

Here is what I tried;

public class GetWs {

   private static String url ="http://somewebservice";
   public static void main(String[] args) {

    InputStream in;
    OutputStream out;
    try {
          out = new FileOutputStream("testoutfile.txt");
          in = new URL(url).openStream();
          int b;
          do {
               b = in.read();
               if (b != -1) {
            out.write(b);
                 out.flush();
               }
           } while (b != -1);
            in.close();out.close();     
    } catch (Exception e) {
        e.printStackTrace();
     }

   }

}

Answer 1

Try compression and streaming to the file output stream, preferrably using NIO.

IF you have to parse and validate the XML, try a STAX parser.