Hello Everyone!
Thank you for reading about this problem I have been trying to solve - I look forward to your responses!
Big picture:
SQL Server 2008 R2, SP 1, running on Windows 2008 R2
Performance behavior is extremely different to two servers, although the resources and load on each server is very similar.
I'm not the DBA of these systems, so I can only get so much information, but I can ask for specific things or make suggestions.
I have written a couple of stored procedures which pull some rows from a table in a linked Oracle server, store the data locally in a permanent table, and then perform some updates and deletes to get the data ready to be imported into a system using an interface to a system's API.
The two SQL Server Database servers are DEV and QA.
In DEV, everthing works great.
The stored procedure I am having a problem with in QA does this (all local on the SQL Server):
1. Inserts selected rows from a staging table into a working table
2. Updates the working table to provide a DEPARTMENT NAME for a column of every row. This is a big UPDATE statement, joining the working table to 2 other tables and using a CASE statement to come up with the DEPARTMENT NAME.
3. Inserts final formatted data into a table to be read by the interface.
The problem in QA is step 2. As I said, the same statement runs fine in DEV.
Here are the details which I can't understand how to fix:
Assume my procedure is -
create procedure SP_FORMAT_DATA as
begin
insert into working (col1, col2 ... coln) select col1, col2, ... coln where conditions ...
update working set department = case when ...
other DML statements
end;
go
In DEV, the procedure completes in 9 seconds.
In QA, the procedure completes in just under 20 minutes. (The update takes most of that time. Everything else completes in a few seconds.)
Now, here is what I think is the key information:
I can get the same work done in QA in 9 seconds by running the SQL statements in separate batches like this:
insert into working (col1, col2 ... coln) select col1, col2, ... coln where conditions ...
GO
update working set department = case when ...
GO
other DML statements
GO
At first I thought the problem was related to the fact that the SQL was running from a procedure, but it seems more tied to the fact that running in one batch is bad in QA, where as there is no negative impact in DEV. For example, even outside of a procedure the following takes 20 minutes in QA (compared to 9 seconds as explained above.):
insert into working (col1, col2 ... coln) select col1, col2, ... coln where conditions ...
update working set department = case when ...
other DML statements
GO
I've compared every setting I know to check between the two servers and the only differences are:
| QA | fill factor (%) | 75 | |
| DEV | fill factor (%) | 0 |
| QA | remote query timeout (s) | 600 | |
| DEV | remote query timeout (s) | 30000 |
| QA | Ole Automation Procedures | 0 | |
| DEV | Ole Automation Procedures | 1 |
Database Settings are identical with the exception of DEV having autoupdate statistics set to true while this is false for QA. I have inserted update statistics commands for all impacted tables in the batch/procedure and that did not help, so I don't think this setting in the problem. (stats collection of the tables only took a few seconds, so updating stats did not take so long as to hide the improvement made by updating them.)
Both databases have these settings:
| textsize | 2147483647 |
| language | us_english |
| dateformat | mdy |
| datefirst | 7 |
| lock_timeout | -1 |
| quoted_identifier | SET |
| arithabort | SET |
| ansi_null_dflt_on | SET |
| ansi_warnings | SET |
| ansi_padding | SET |
| ansi_nulls | SET |
| concat_null_yields_null | SET |
| isolation level | read committed snapshot |
This problem truly has me stumped. Any suggestions? Is this related to how the servers may handle client connections differently? I can get on the server myself and see what the configuration might be, but I can ask the DBA to check somethings if you give me some questions for them.
Also, a work around would be to emulate separate batches from within the stored procedure, so a suggestion on how to do that would be good. I guess I could break this procedure up into 2 or 3 smaller ones, which I may end up doing if I have to, but it sure would be better if I could get QA to work like DEV. Other folks are having problems with interfaces in QA as well, so maybe the solution to my problem will fix many other problems.
Thanks!