Question

I have a problem with the Google App Engine JDO implementation that I cannot figure out. The documentation (http://code.google.com/intl/sv-SE/appengine/docs/java/datastore/jdo/creatinggettinganddeletingdata.html) states “The call to makePersistent() is synchronous, and doesn’t return until the object is saved and indexes are updated.” but my experience differs.

I want to save (makePersistent) an object into the datastore. When the save is complete I want to be able to fetch it (Query execute) from the datastore immediately. I know that I don’t have to fetch it (because I already have the object in memory) but the point is that I want the next request to be able to retrieve the data from the datastore. That is not working with the current implementation if the second request is fast enough.

One strange thing I’ve noticed is that if I’m trying to fetch the object from the datastore a couple of times in a loop (code below) the object is returned really quick (usually < 10ms). But if I skip the loop and instead run Thread.sleep(..) for 5000 ms between makePersistent and query execute it’s not certain that the object is found. None of these solutions are what I want. I want to be able to fetch the data right away without a sleep or loop in between.

The code and the result from accessing the DataStoreTestServlet below is as you can see including the loop that is “waiting” for the data to be found. Again, I don’t want the loop.

Does anyone know what I’m missing? I guess it has to be something. This implementation doesn’t feel right to me :).

I’m using appengine-java-sdk-1.6.0. This is an issue both locally (development server) and when deployed on the Google servers.

Here’s the result from accessing the servlet.

Created users:
User [password=password, userName=user1321190966416] took 18ms, 2 loop(s)
User [password=password, userName=user1321190966438] took 15ms, 6 loop(s)
User [password=password, userName=user1321190966456] took 2ms, 1 loop(s)
User [password=password, userName=user1321190966460] took 10ms, 5 loop(s)
User [password=password, userName=user1321190966472] took 0ms, 1 loop(s)
User [password=password, userName=user1321190966472] took 0ms, 1 loop(s)
User [password=password, userName=user1321190966472] took 16ms, 1 loop(s)
User [password=password, userName=user1321190966488] took 0ms, 2 loop(s)
User [password=password, userName=user1321190966488] took 0ms, 1 loop(s)
User [password=password, userName=user1321190966488] took 16ms, 1 loop(s)

The code and configuration.

jdoconfig.xml

<?xml version="1.0" encoding="utf-8"?>
<jdoconfig xmlns="http://java.sun.com/xml/ns/jdo/jdoconfig" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://java.sun.com/xml/ns/jdo/jdoconfig">
<persistence-manager-factory name="transactions-optional">
<property name="javax.jdo.PersistenceManagerFactoryClass" value="org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory"/>
<property name="javax.jdo.option.ConnectionURL" value="appengine"/>
<property name="javax.jdo.option.NontransactionalRead" value="true"/>
<property name="javax.jdo.option.NontransactionalWrite" value="true"/>
<property name="javax.jdo.option.RetainValues" value="true"/>
<property name="datanucleus.appengine.autoCreateDatastoreTxns" value="true"/>
<property name="datanucleus.appengine.datastoreReadConsistency" value="STRONG" />
</persistence-manager-factory>
</jdoconfig>

PMF.java

import javax.jdo.JDOHelper;
import javax.jdo.PersistenceManagerFactory;

public final class PMF {
    private static final PersistenceManagerFactory pmfInstance = JDOHelper.getPersistenceManagerFactory("transactions-optional");

    private PMF() {
    }

    public static PersistenceManagerFactory get() {
        return pmfInstance;
    }
}

User.java

import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class User {
    @PrimaryKey
    @Persistent
    private String userName;

    @Persistent
    private String password;

    public User(String userName, String password) {
        super();
        this.setUserName(userName);
        this.setPassword(password);
    }

    public String getUserName() {
        return userName;
    }

    public String getPassword() {
        return password;
    }

    public void setUserName(String userName) {
        this.userName = userName;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public String toString() {
        return "User [password=" + password + ", userName=" + userName + "]";
    }
}

DataStoreTestServlet.java

import java.io.IOException;
import java.util.List;
import javax.jdo.PersistenceManager;
import javax.jdo.Query;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
public class DataStoreTestServlet extends HttpServlet {
    public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
        StringBuffer sb = new StringBuffer();
        sb.append("Created users:\n");

        for (int i = 0; i < 10; i++) {
            String uniqueName = "user" + System.currentTimeMillis();
            User user = new User(uniqueName, "password");
            save(user);

            User userFromDS = null;
            long startTime = System.currentTimeMillis();
            long loop = 0;
            while (userFromDS == null) {
                userFromDS = get(uniqueName);
                loop++;
                if (userFromDS != null) {
                    long endTime = System.currentTimeMillis();
                    sb.append(userFromDS.toString() + " took " + (endTime - startTime) + "ms, " + loop + " loop(s)\n");
                }
            }
        }
        resp.setContentType("text/plain");
        resp.getWriter().println(sb.toString());
    }

    public Object save(Object obj) {
        PersistenceManager pm = PMF.get().getPersistenceManager();
        Object savedObject = null;
        try {
            savedObject = pm.makePersistent(obj);
        } finally {
            pm.close();
        }
        return savedObject;
    }

    public User get(String userName) {
        User user = null;
        List<User> users = null;
        PersistenceManager pm = PMF.get().getPersistenceManager();
        Query query = pm.newQuery(User.class);
        query.setFilter("userName == nameParam");
        query.declareParameters("String nameParam");
        try {
            users = (List<User>) query.execute(userName);
            if (users != null && users.size() > 0) {
                user = users.get(0);
            }
        } finally {
            query.closeAll();
            pm.close();
        }
        return user;
    }
}

Answer 1

Try adding this in your jdoconfig.xml:

<property name="datanucleus.appengine.datastoreReadConsistency" value="STRONG" />

To increase performance; the app engine datastore is “eventually consistent”. This means that when you make new objects or alter existing ones, it doesn’t show up right away; allowing for performance gains in lookup times. The opposite of this is “STRONG” consistency, meaning every request is made using the most recent data in the datastore.

Now, according to the app engine documentation for this STRONG consistent is the default, and you have to explicitly set eventual consistency. But, from what I’ve observed, you have to set STRONG consistency and EVENTUAL is the default (bug maybe?). So try adding the above to your jdoconfig xml and if you observe the same thing I did then I’ll probably open a bug against app engine, assuming one hasn’t been opened already for this issue.

The one thing you’re going to have to keep in mind is that if you set STRONG consistency, you’re going to take a performance hit. I only set it because there were parts of my interface that were getting messed up because I would build a piece of it with not so fresh data, and during the same request another piece would get built with fresh data; making my interface inconsistent. This might be a broad approach to fixing the problem; but it works :).