Hi,
There are latch wait errors in our Sharepoint SQL database server logs. This is a SQL 2005 SP4 server on Windows 2003 R2 SP2 (64 bit) active/passive cluster with around 32GB RAM . There are no IO errors or any other errors other than the latch timeout errors. Server is working as expected and wait stats does not indicate anything suspicious. I understand that this error occurred while defragmenting index. Is there any optimization that I can do to prevent such errors? Any advice other than reaching out to Microsoft support?
Thanks in advance
SQL Errorlog
2012-06-18 19:47:50.04 spid948 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_VIRTUAL_ROOT', id 000000022D7A8EA8, type 2, Task 0x00000000891BD438 : 2, waittime 300, flags 0x1a, owning task 0x0000000495303D68.
Continuing to wait.
2012-06-18 19:47:50.04 spid948 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_VIRTUAL_ROOT', id 000000022D7A8EA8, type 2, Task 0x0000000003C272E8 : 15, waittime 300, flags 0x1a, owning task 0x0000000495303D68. Continuing
to wait.
2012-06-18 19:47:50.04 spid948 A time-out occurred while waiting for buffer latch -- type 2, bp 00000001A7FEED00, page 1:3718949, stat 0x8c0000b, database id: 41, allocation unit Id: 72057612347899904, task 0x0000000003C816D8 : 13, waittime
300, flags 0x1a, owning task 0x0000000003C949B8. Not continuing to wait.
2012-06-18 19:47:50.04 spid948 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_VIRTUAL_ROOT', id 000000022D7A8EA8, type 2, Task 0x00000005834F25C8 : 4, waittime 300, flags 0x1a, owning task 0x0000000495303D68. Continuing
to wait.
2012-06-18 19:47:50.04 spid948 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_VIRTUAL_ROOT', id 000000022D7A8EA8, type 2, Task 0x0000000003C63C18 : 9, waittime 300, flags 0x1a, owning task 0x0000000495303D68. Continuing
to wait.
2012-06-18 19:47:50.04 spid948 A time-out occurred while waiting for buffer latch -- type 4, bp 0000000604FF0700, page 1:598, stat 0xac0000b, database id: 41, allocation unit Id: 72057612347899904, task 0x0000000495303D68 : 1, waittime
300, flags 0x1018, owning task 0x0000000000000000. Not continuing to wait.
2012-06-18 19:47:50.05 spid948 Using 'dbghelp.dll' version '4.0.5'
2012-06-18 19:47:50.05 spid948 A time-out occurred while waiting for buffer latch -- type 2, bp 00000005D7FC2C00, page 1:3720107, stat 0xbc0000b, database id: 41, allocation unit Id: 72057612347965440, task 0x0000000003C3B828 : 16, waittime
300, flags 0x1a, owning task 0x0000000003C8B588. Not continuing to wait.
2012-06-18 19:47:50.05 spid948 A time-out occurred while waiting for buffer latch -- type 4, bp 00000001A8FE7A80, page 1:3761345, stat 0xde8800b, database id: 41, allocation unit Id: 72057612347965440, task 0x0000000003C949B8 : 12, waittime
300, flags 0x200000001a, owning task 0x0000000003C8B588. Not continuing to wait.
2012-06-18 19:47:50.09 spid948 **Dump thread - spid = 948, PSS = 0x00000005B9095A80, EC = 0x00000005821041C0
2012-06-18 19:47:50.10 spid948 ***Stack Dump being sent to S:\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0008.txt
2012-06-18 19:47:50.10 spid948 * *******************************************************************************
2012-06-18 19:47:50.10 spid948 *
2012-06-18 19:47:50.10 spid948 * BEGIN STACK DUMP:
2012-06-18 19:47:50.10 spid948 * 06/18/12 19:47:50 spid 948
2012-06-18 19:47:50.10 spid948 *
2012-06-18 19:47:50.10 spid948 * Latch timeout
2012-06-18 19:47:50.11 spid948 *
2012-06-18 19:47:50.11 spid948 * Input Buffer 78 bytes -
2012-06-18 19:47:50.11 spid948 * 16 00 00 00 12 00 00 00 02 00 00 00 00 00 00 00 00 00
2012-06-18 19:47:50.11 spid948 * d b o . p r 01 00 00 00 1a 00 64 00 62 00 6f 00 2e 00 70 00 72 00
2012-06-18 19:47:50.11 spid948 * o c _ D e f r a g 6f 00 63 00 5f 00 44 00 65 00 66 00 72 00 61 00 67 00
2012-06-18 19:47:50.11 spid948 * m e n t I n d i c 6d 00 65 00 6e 00 74 00 49 00 6e 00 64 00 69 00 63 00
2012-06-18 19:47:50.11 spid948 * e s 65 00 73 00 00 00
2012-06-18 19:47:50.11 spid948 *
2012-06-18 19:47:50.11 spid948 * *******************************************************************************
2012-06-18 19:47:50.11 spid948 * -------------------------------------------------------------------------------
2012-06-18 19:47:50.11 spid948 * Short Stack Dump
2012-06-18 19:47:50.16 spid948 Stack Signature for the dump is 0x00000000DF7CF376
Output from debugging the Dumpfile:
ntdll!ZwWaitForSingleObject+0xa
kernel32!WaitForSingleObjectEx+0x130
sqlservr!CDmpDump::DumpInternal+0x4a3
sqlservr!CDmpDump::DumpFilter+0xbc
sqlservr!CDmpDump::Dump+0xb1
msvcr80!_C_specific_handler+0x96
ntdll!RtlpExecuteHandlerForException+0xd
ntdll!RtlDispatchException+0x1b4
ntdll!RtlRaiseException+0xae
kernel32!RaiseException+0x73
sqlservr!CDmpDump::Dump+0x7c
sqlservr!CImageHelper::DoMiniDump+0x413
sqlservr!stackTrace+0x6e7
sqlservr!LatchBase::DumpOnTimeoutIfNeeded+0x1a0
sqlservr!LatchBase::PrintWarning+0x1de
sqlservr!alloca_probe+0x7ff34
sqlservr!BTreeMgr::Seek+0xfa1
sqlservr!BTreeMgr::GetHPageIdWithKey+0x19b
sqlservr!IndexPageManager::GetPageForLinkModification+0x122
sqlservr!LocatePageForSplit+0x66
sqlservr!SplitPage+0x1b4
sqlservr!IndexDataSetSession::InsertSmallRecord+0xbc2
sqlservr!IndexDataSetSession::InsertRow+0x40e
sqlservr!IndexDataSetSession::InsertRowFromOIB+0x2d
sqlservr!RowsetNewSS::InsertRowFromOIB+0x2b
sqlservr!CValRowCrtIdx::SetDataX+0xfd
sqlservr!CEs::GeneralEval+0x21cc
sqlservr!CQScanOnlineIndexNew::NonOptimizedGetAndInsertRows+0x184
sqlservr!CQScanOnlineIndexNew::GetRow+0x56
sqlservr!CQScanXProducerNew::ProcessRows<0>+0x4a
sqlservr!CQScanXProducerNew::Open+0xbc
sqlservr!FnProducerOpen+0x43
sqlservr!FnProducerThread+0x3b7
sqlservr!SubprocEntrypoint+0x4e1
sqlservr!SOS_Task::Param::Execute+0xee
sqlservr!SOS_Scheduler::RunTask+0xc9
sqlservr!SOS_Scheduler::ProcessTasks+0xb4
sqlservr!SchedulerManager::WorkerEntryPoint+0xe7
sqlservr!SystemThread::RunWorker+0x59
sqlservr!SystemThreadDispatcher::ProcessWorker+0x130
sqlservr!SchedulerManager::ThreadEntryPoint+0x128
msvcr80!endthreadex+0x47
msvcr80!endthreadex+0x104
kernel32!BaseThreadStart+0x3a
Wait stats
wait_type | wait_time_s | Percentage |
BACKUPIO | 8006804.05 | 20.53 |
CXPACKET | 6599764.81 | 16.92 |
BACKUPBUFFER | 6437860.14 | 16.51 |
ASYNC_IO_COMPLETION | 6432572.16 | 16.49 |
ASYNC_NETWORK_IO | 2687123.39 | 6.89 |
SOS_SCHEDULER_YIELD | 2538972.67 | 6.51 |
OLEDB | 2078496.17 | 5.33 |
LATCH_EX | 1974265.97 | 5.06 |
BACKUPTHREAD | 1250778.97 | 3.21 |
Top 10 latch waits from sys.dm_os_latch_stats
latch_class | waiting_requests_count | wait_time_ms | max_wait_time_ms |
DBCC_MULTIOBJECT_SCANNER | 219166469 | 810446217 | 33390 |
ACCESS_METHODS_SCAN_RANGE_GENERATOR | 68449330 | 438454324 | 1719 |
DBCC_FILE_CHECK_OBJECT | 133646801 | 388468501 | 1656 |
NESTING_TRANSACTION_FULL | 51068563 | 198654802 | 3266 |
BUFFER | 342778059 | 122574651 | 300015 |
DBCC_CHECK_AGGREGATE | 15101518 | 85256332 | 1015 |
ACCESS_METHODS_HOBT_VIRTUAL_ROOT | 7156670 | 53577646 | 306016 |
ACCESS_METHODS_DATASET_PARENT | 814522 | 36186448 | 9860 |
NESTING_TRANSACTION_READONOY | 16407896 | 6881130 | 875 |
ACCESS_METHODS_ACCESSOR_CACHE | 448885 | 1021539 | 2015 |