Question

I need to know if it is standard practice to decompose complex queries into parts and create temporary tables which are dropped at the end.? In OLAP applications it shouldnt be much of an issue, but in OLTP since speed matters is it avoided?.

Answer 1

For simple queries which are well-optimized by your DBMS, temporary tables are usually a bad idea because they introduce overhead.

But sometimes your DBMS will have a really hard time optimizing complex queries. At that point you have at least 5 options:

1. change your schema or indexes to make it easier for the optimizer to choose a better query plan
2. tweak your SQL to get the DBMS to choose the indexes, join strategies, etc. that you want and to work around known and unknown bugs in your DBMS’s optimizer.
3. use “hints” to get the DBMS to choose the indexes, join strategies, etc. that you want.
4. Get the plan you want and use a “saved plan” to force its use by the DBMS.
5. use temp tables (or table variables, etc.) to decompose complex queries into simpler intermediate queries

There’s no hard-and-fast rule about which option is best for any particular query. I’ve used all of the above strategies. I tend to choose the temp table approach when I don’t own the schema, so I can’t change it, and when I don’t want to depend on hints or query tuning or saved plans (often because I don’t want to expose myself to changes in the underlying schema made later).

Keep in mind that using temp tables to decompose queries will give you sub-optimal performance every time. But it’s usually predictably sub-optimal. The worst case using temp tables isn’t nearly as bad as when your DBMS chooses a bad plan for a single large query. This happens surprisingly often, especially in the face of changes in underlying schema, DBMS version changes, dev vs. production differences, etc.

Personally, I find that if a query gets to a level of complexity where I have to bend over backwards to get the DBMS to do what I want, and if I feel that maintainability of the application is at risk, then I’ll often go with decomposition and temp tables if I can’t change the schema or indexes.

Of course, in theory you shouldn’t be running expensive, complex queries on your OLTP database, but in practice most applications are never “pure” OLTP– there’s always a few complicated, hard-to-optimize queries in any OLTP project.