I have run a weird problem. Maybe someone here can help me.
My SQL Server version is 2008 R2. It runs at a server with 24 cores.
I have 2 query strings:
String SQL_1 = "select
t.testconfig_id, t.minuteSequence, t.location_id,
sum(t.vuPerNode) as totalVu,
sum(t.backOffPctSum) / sum(t.recordNum) as avgBackOffPct
from
(select
p.testconfig_id, p.minuteSequence, r.location_Id,
SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode,
SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum
from
loadtest_progress_in_minute p (nolock)
join
loadtestRunrecord r (nolock) on p.test_id = r.test_id and p.nodeId = r.nodeId
where
p.test_id = ?
group by
p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id) t
group by
t.testconfig_id, t.minuteSequence, t.location_id
order by
t.testconfig_id, t.minuteSequence, t.location_id option (maxdop 23)"
String SQL-2 = "select
t.testconfig_id, t.minuteSequence, t.location_id,
sum(t.vuPerNode) as totalVu,
sum(t.backOffPctSum) / sum(t.recordNum) as avgBackOffPct
from
(select
p.testconfig_id, p.minuteSequence, r.location_Id,
SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode,
SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum
from
loadtest_progress_in_minute p (nolock)
join
loadtestRunrecord r (nolock) on p.test_id = r.test_id and p.nodeId = r.nodeId
where
p.test_id = ?
group by
p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id) t
group by
t.testconfig_id, t.minuteSequence, t.location_id
order by
t.testconfig_id, t.minuteSequence, t.location_id option (maxdop 23) "
The only difference between the two queries is that SQL-2 has one extra tab character in the end.
I run these 2 queries with the following code in the same environment:
PreparedStatemen ps = conn.prepareStatement(SQL_1);
//PreparedStatemen ps = conn.prepareStatement(SQL_2);
ps.setLong(1234);
ps.execute();
I found the performance of the 2 queries is very different in the code snippet sometimes.
ps.excute() of SQL_1 in this code snippet only costs about 10 seconds. I see the CPU usage is high when SQL_1 is running. 24 CPU of the server is all utilized.
But ps.excute() of SQL_2 in the same code snippet costs about 150 seconds. The CPU usage is slow when SQL_2 is running. Only 2 of 24 CPU are utilized.
SQL_1 and SQL_2 are running at the same time.
But the observation above is not always so. Sometime ps.execute() performance of SQL_1 and SQL_2 are same. Sometime ps.execute() performance of SQL_1 and SQL_2 are as what I described above.
That’s what I found. It’s very confusing. Last whitespace of SQL would hurt SQL Server performance?
I think it is not caused by auto-parameterization cache as described in
Space in SQL Server 2008 R2 slows down performance
Because I got the above observation by invoking the code snippet in an infinite loop for long time.
From wireshark I found Microsoft JDBC would append 2 extra tab(1 tab character = 4 space character) characters between the SQL String and Parameter like following:
[20] [00] [20] [00] [20] [00] [20] [00] [20] [00] [20] [00][20] [00][20] [00]
I don’t know whether it is related to my problem.
Thanks.
Updates 2012-8-20:
I executed the 3 following sql in sql-server management studio. They have the same execution plan. But I have weird findings:
declare @sql varchar(max);
set @sql='Declare @testId bigint;set @testId = 1234;select p.testconfig_id, p.minuteSequence,r.location_Id, SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode, SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum from loadtest_progress_in_minute p with( nolock,index(idx_loadtest_progress_in_minute_1) ) join loadtestRunrecord r ( nolock ) on p.test_id = r.test_id and p.nodeId = r.nodeId where p.test_id = @testId group by p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id option (maxdop 23)';
execute (@sql);
This sql’s performance is very bad. It takes about 90 seconds. Only 2 CPU are used.
Declare @sSQL nvarchar(2000);
Declare @paramDefine nvarchar(2000);
Declare @testId bigint;
set @testId = 1234;
set @sSQL = N' select t.testconfig_id, t.minuteSequence, t.location_id, sum(t.vuPerNode) as totalVu, sum(t.backOffPctSum) / sum(t.recordNum) as avgBackOffPct from ( select p.testconfig_id, p.minuteSequence, r.location_Id, SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode, SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum from loadtest_progress_in_minute p ( nolock ) join loadtestRunrecord r ( nolock ) on p.test_id = r.test_id and p.nodeId = r.nodeId where p.test_id = @P0 group by p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id ) t group by t.testconfig_id, t.minuteSequence, t.location_id order by t.testconfig_id, t.minuteSequence, t.location_id option (maxdop 23) ';
set @paramDefine = N'@P0 bigint';
execute sp_executesql @sSQL, @paramDefine, @P0 = @testId;
This sql takes only about 10 seconds in management studio. All CPU are used.
declare @p1 int
--set @p1=1
exec sp_prepexec @p1 output,N'@P0 bigint',N' select t.testconfig_id, t.minuteSequence, t.location_id, sum(t.vuPerNode) as totalVu, sum(t.backOffPctSum) / sum(t.recordNum) as avgBackOffPct from ( select p.testconfig_id, p.minuteSequence, r.location_Id, SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode, SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum from loadtest_progress_in_minute p ( nolock ) join loadtestRunrecord r ( nolock ) on p.test_id = r.test_id and p.nodeId = r.nodeId where p.test_id = @P0 group by p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id ) t group by t.testconfig_id, t.minuteSequence, t.location_id order by t.testconfig_id, t.minuteSequence, t.location_id option (maxdop 23) ',@P0=1234
select @p1
This sql performance is fast. Only takes about 10 seconds. All CPU are utilized.
Update 2012-8-21
Until now, I have not get a final clear conclusion about what I have found. Because windows world is not open, maybe we never get the detail inside SQL Server. Here I just give my explaination about what I have found. Some explaination is just my guess. I hope that it’s helpful for others.
1) why I get the different performance for two similiar SQL in JDBC sometimes(These SQL are the same except whether the last tab character is present)
Our test is not in an isolated environment. When I am test the two SQL with JDBC, other processes also excute the SQL with the last tab character is present at the same time. So our test result is influenced by other processes.
The different performance’s root cause is that they choose the different excution plan. One chose the exection plan with good parellesim. The other one chose the execution plan with bad parellesim. Because all the other processes are executing SQL with tab character, SQL without tab character is treated as a new SQL. So when SQL without tab character is executed, it generates a new execution plan based on typical parameter value according to records statistics. Maybe the typical value is not good at performance at first time. But actual visited parameter value histogram can flush the execution plan cache. The SQL without tab character is only used in my test. My test only use parameter value (1234). SQL server thinks that (1234) is often visited for the SQL and flush the execution plan with the actual most common visited parameter value (1234). So the performance become good.
When I switch back to the SQL with tab character, SQL Server would take a older execution plan cache. This cache can be introduced by other running processes and influenced by others. This execution plan is also generated based on actual most popular visited parameter value. But this value is influenced by other processes. So it may not be (1234) and execution plan based on the value is not good for (1234) at performance. That’s why SQL performance with tab character is bad sometimes.
Because sometimes my test program for SQL with tab character also can flush excution plan cache if my test is running frequently enough to change the actual visited parameter value . The performance of SQL with tab character also can become good sometimes.
2) why the following SQL in SSMS is alway slow
declare @sql varchar(max)
set @sql='Declare @testId bigint;set @testId = 36887;select p.testconfig_id, p.minuteSequence,r.location_Id, SUM(p.activeCount) * 1.0 / COUNT(1) as vuPerNode, SUM(p.backOffPct) as backOffPctSum, COUNT(1) as recordNum from loadtest_progress_in_minute p with( nolock,index(idx_loadtest_progress_in_minute_1) ) join loadtestRunrecord r ( nolock ) on p.test_id = r.test_id and p.nodeId = r.nodeId where p.test_id = @testId group by p.testconfig_id, p.minuteSequence, p.nodeId, r.location_id option (maxdop 23)'
execute (@sql)
Because SSMS also generates execution plan based on typical parameter value of records statistics. The parameter value (1234) is atypical value. So the above SQL is slow at the beginning. I guess that ‘execute’ command in SSMS is special and its cache won’t be flushed by actual most common visited parameter value. So it is always slow. According to my experiment results, I guess ‘sp_prepexec’ and ‘sp_executesql’ are different from ‘execute’ and their plan cache can be flushed by actual most common visited parameter value and have a similiar behavior with JDBC.
3) why adding recompile hint would speed up the above SQL’s performance
Before answering this problem, let’s take a look at the following text from MSDN online help document.
“Instructs the SQL Server Database Engine to discard the plan generated for the query after it executes, forcing the query optimizer to recompile a query plan the next time the same query is executed. Without specifying RECOMPILE, the Database Engine caches query plans and reuses them. When compiling query plans, the RECOMPILE query hint uses the current values of any local variables in the query and, if the query is inside a stored procedure, the current values passed to any parameters.
RECOMPILE is a useful alternative to creating a stored procedure that uses the WITH RECOMPILE clause when only a subset of queries inside the stored procedure, instead of the whole stored procedure, must be recompiled. For more information, see Recompile a Stored Procedure. RECOMPILE is also useful when you create plan guides.”
Please note the following sentence :
When compiling query plans, the RECOMPILE query hint uses the current values of any local variables in the query and, if the query is inside a stored procedure, the current values passed to any parameters.
It means that RECOMPILE query hint change the SSMS execution plan generation behavior. SSMS generates execution plan based on typical parameter value witout RECOMPILE query hint while SSMS generates execution paln based on current parameter value with RECOMPILE query hint. So recompile hint make execution plan is perfect for current parameter value (1234)
In one word, execution plan is chosen by complex factors. We must consider it carefully.
First, the white space at the end of the query would not make a difference. It is inconceivable that white space could cause a performance problem in any database, other than lengthening the amount of time for transmitting and parsing the statement. The SQL Engine reads the query during the compile phase and produces an execution plan. The execution plan is what gets run. Extra white space is lost in the very first step of tokenizing the query string. As far as I know, all database engines work this way.
When testing performance of queries, you need to deal with the number one cause of variability in performance: caching. The second time you run a query, it will usually go much faster, because the tables are already in the page cache.
One way is to clear the cache between runs. Another is to run the query multiple times, and ignore the first run.
In any case, your first query is not proper syntax, so that might have something to do with what you are seeing. The select statement is:
The group by is:
The variables t.id1, t.sequence, and t.id2 should be causing a compile-time error in SQL Server or any reasonable database because they are neither in aggregation functions nor in the group by clause (this is a friendly dig at the hidden columns in MySQL, which would allow this syntax).