Question

On development server, I am trying to write data to a single file using FileService api.
Even if I write small amount of data, closing the file (with close()) doesn’t free memory.
I need to use closeFinally() from FileService to free memory. Therefore my guess is that the entire file is being cached in memory until closeFinally() has been call, which can cause some problems if I want to write a huge file in a single blob.
Is there any way to solve this memory issue ?

Here is the sample code :

String veryLongString = "verryyyyllooooonnnnggggssssttttrrrriiiiinnnnnggggg";
for (int i = 0; i < 10; ++i) {
    veryLongString += veryLongString;
}
// veryLongString.length() => 51200 char

FileService FILESERVICE =
    FileServiceFactory.getFileService();

// Create new file
String newPath =
    FILESERVICE.createNewBlobFile("application/octet-stream").getFullPath();

for (int i = 0; i < 10; ++i) {
    System.out.println(i);

    // Open file for append
    AppEngineFile file = new AppEngineFile(newPath);
    FileWriteChannel writeChannel = FILESERVICE.openWriteChannel(file, true);
    OutputStream out = Channels.newOutputStream(writeChannel);

    // Append some data to this file
    for (int j = 0; j < 5000; ++j) {
        out.write(veryLongString.getBytes());
    }
    writeChannel.close();
    // Writed 5000*51200 => 256Mo char (to memory, should be
    // also persisted to disk)
}

// Writed all data we want, we finally close the file
AppEngineFile file = new AppEngineFile(newPath);
FileWriteChannel writeChannel = FILESERVICE.openWriteChannel(file, true);
writeChannel.closeFinally();

Output :

0
1
2
3
com.google.apphosting.api.ApiProxy$UnknownException: An error occurred for the API request file.Append().
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:518)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:452)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.call(ApiProxyLocalImpl.java:430)
    at java.util.concurrent.Executors$PrivilegedCallable$1.run(Executors.java:463)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.util.concurrent.Executors$PrivilegedCallable.call(Executors.java:460)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2786)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
    at java.io.OutputStream.write(OutputStream.java:58)
    at com.google.appengine.api.files.dev.FileMetadata.append(FileMetadata.java:198)
    at com.google.appengine.api.files.dev.LocalFileService.append(LocalFileService.java:357)
    at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.google.appengine.tools.development.ApiProxyLocalImpl$AsyncApiCall.callInternal(ApiProxyLocalImpl.java:498)
    ... 10 more

Answer 1

The development server is designed to help you test and debug your application; scalability is not a primary concern for it. You’re correct that it caches the file in memory; this is how it’s supposed to work.