Question

I was wondering when dealing with a web service API that returns XML, whether it’s better (faster) to just call the external service each time and parse the XML (using ElementTree) for display on your site or to save the records into the database (after parsing it once or however many times you need to each day) and make database calls instead for that same information.

Answer 1

Everyone is being very polite in answering this question: “it depends”… “you should test”… and so forth.

True, the question does not go into great detail about the application and network topographies involved, but if the question is even being asked, then it’s likely a) the DB is “local” to the application (on the same subnet, or the same machine, or in memory), and b) the webservice is not. After all, the OP uses the phrases “external service” and “display on your own site.” The phrase “parsing it once or however many times you need to each day” also suggests a set of data that doesn’t exactly change every second.

The classic SOA myth is that the network is always available; going a step further, I’d say it’s a myth that the network is always available with low latency. Unless your own internal systems are crap, sending an HTTP query across the Internet will always be slower than a query to a local DB or DB cluster. There are any number of reasons for this: number of hops to the remote server, outage or degradation issues that you can’t control on the remote end, and the internal processing time for the remote web service application to analyze your request, hit its own persistence backend (aka DB), and return a result.

Fire up your app. Do some latency and response times to your DB. Now do the same to a remote web service. Unless your DB is also across the Internet, you’ll notice a huge difference.

It’s not at all hard for a competent technologist to scale a DB, or for you to completely remove the DB from caching using memcached and other paradigms; the latency between servers sitting near each other in the datacentre is monumentally less than between machines over the Internet (and more secure, to boot). Even if achieving this scale requires some thought, it’s under your control, unlike a remote web service whose scaling and latency are totally opaque to you. I, for one, would not be too happy with the idea that the availability and responsiveness of my site are based on someone else entirely.

Finally, what happens if the remote web service is unavailable? Imagine a world where every request to your site involves a request over the Internet to some other site. What happens if that other site is unavailable? Do your users watch a spinning cursor of death for several hours? Do they enjoy an Error 500 while your site borks on this unexpected external dependency?

If you find yourself adopting an architecture whose fundamental features depend on a remote Internet call for every request, think very carefully about your application before deciding if you can live with the consequences.