Question

I’m trying to use the free Google App Engine as a backend for Google Cloud Messages for my next Android app but when I have “finished” writing the server it already uses almost 100% of the free frontend instance hours. The question I have is if and how I can improve this?

The application is a servlet that is called every 15 minutes from a cron job, the servlet downloads and parses 3 RSS feeds and checks if anything has changed since the last call, saves the dates to the database (JDO and memcache, 3 calls) to know when the last running was and if any changes have happend since the last call sends that information out the the connected phones, right now 3 phones are connected, it’s just one call to Googles servers. No data is returned from the servlet.

Here is the code

public void doGet(HttpServletRequest request, HttpServletResponse response)
    throws IOException
{
    boolean sendMessage = false;

    String eventsFeedUrl = "http://rss.com";
    String newsFeedUrl = "http://rss2.com";
    String trafficFeedUrl = "http://rss3.com";

    response.setContentType("text/plain");
    Message.Builder messageBuilder = new Message.Builder();
    String messageData = getFeedMessageData(eventsFeedUrl);
    if (!messageData.equals(StringUtils.EMPTY))
    {
        messageBuilder.addData("event", messageData);
        sendMessage = true;
    }
    messageData = getFeedMessageData(newsFeedUrl);
    if (!messageData.equals(StringUtils.EMPTY))
    {
        messageBuilder.addData("news", messageData);
        sendMessage = true;
    }
    messageData = getFeedMessageData(trafficFeedUrl);
    if (!messageData.equals(StringUtils.EMPTY))
    {
        messageBuilder.addData("traffic", messageData);
        sendMessage = true;
    }
    if (sendMessage)
    {
        sendMessage(messageBuilder.build(), response, debug);
    }
}

private void sendMessage(Message message, HttpServletResponse response, boolean debug)
    throws IOException
{
    SendResult sendResult = GCMService.send(message, Device.list());
    int deleteCount = 0;
    for (MessageResult errorResult : sendResult.getErrorResults())
    {
        if (deleteCount < 200 && (errorResult.getErrorName().equals(Constants.ERROR_NOT_REGISTERED) || errorResult.getErrorName().equals(Constants.ERROR_INVALID_REGISTRATION)))
        {
            Device.delete(errorResult.getDeviceId());
            deleteCount++;
        }
    }
}

private String getFeedMessageData(String feedUrl)
{
    String messageData = StringUtils.EMPTY;
    FeedHistory history = FeedHistory.getFeedHistoryItem(feedUrl);
    Feed feedContent = RssParser.parse(feedUrl);
    if (feedContent != null && feedContent.getFeedItems().size() > 0)
    {
        if (history == null)
        {
            history = new FeedHistory(feedUrl);
            history.setLastDate(new Date(0));
            history.save();
        }
        for (FeedItem item : feedContent.getFeedItems())
        {
            if (item.getDate().after(history.getLastDate()))
            {
                messageData += "|" + item.getCountyId();
            }
        }
        if (!messageData.equals(StringUtils.EMPTY))
        {
            messageData = new SimpleDateFormat("yyyyMMddHHmmssZ").format(history.getLastDate()) + messageData;
        }
        history.setLastDate(feedContent.getFeedItem(0).getDate());
        history.save();
    }
    return messageData;
}

The call Device.list() uses memcache so after one call it will be cached, the RSS parser is a simple parser that uses org.w3c.dom.NodeList and javax.xml.parsers.DocumentBuilder. According to the log file I use the same instance for days so there are no problems with instances starting up and taking resources. A normal call to the servlet looks like this in the log,

ms=1480 cpu_ms=653 api_cpu_ms=0 cpm_usd=0.019673

I have some ideas of what to try next, try to do the RSS download calls async to minimize the request time. Move the RSS parsing to a backgroud job. What else can be done? It feels like I have done some fundamental errors with my code here because how can a normal web app work if this servlet can’t be called 100 times during 24 hours without consuming 100% of the frontend hours.

/Viktor

Answer 1

Your idle instances hang around for a little while before shutting themselves down. I don’t know how long this is, but I’m guessing it’s somewhere in the 5-15 minute range. If it is in fact 15 minutes, then your cron job hitting it every 15 minutes will keep it alive indefinitely, so you’ll end up using 24 instance hours a day.

You can test this theory by setting your cron job to run every 30 minutes, and see if it halves your instance hour usage.