This is a really odd one. Everytime I encounter these particular waits at a customer site its due to poor configuration or under sized storage units. This is happening in house in a test lab where I can verify everything.
I have 90+ queries that are trying to do truncates, inserts, updates, and deletes and are all waiting on one of the above waits. Perfmon shows no activity on the disk. The disk array shows no activity on its performance monitor. I can even run a quick IO sim with test datafiles on the same LUNs and it works as expected. No Windows errors are showing up in the event log.
Configuration:
- SQL Server 2012 Enterprise
- 2 x 8 Core 2.5Ghz CPUs, blade server
- 128GB RAM, 40GB dedicated to SQL
- Nothing else on box
- Blade is using FCoE to blade chassis and then FC to the SAN
- gobs of RAM on disk array. stripped over 64 15K FC disks
I'm getting the usual spew of crap into the SQL ERRORLOG:
2012-10-04 16:29:44.96 spid24s A time-out occurred while waiting for buffer latch -- type 2, bp 00000020FEF9C240, page 1:139, stat 0x9, database id: 6, allocation unit Id: 458752/281474977169408, task 0x0000001D03E27C38 : 0, waittime 300 seconds, flags 0x3a, owning task 0x0000002056BD6188. Not continuing to wait. 2012-10-04 16:30:48.25 spid8s SQL Server has encountered 14 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [C:\db\tempDB\1\Database24_100_Log_2.ldf] in database [Database_24] (5). The OS file handle is 0x0000000000001C18. The offset of the latest long I/O is: 0x000001d09a8a00 2012-10-04 16:31:43.85 spid8s SQL Server has encountered 5 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\log\Database_24_Stage_log.ldf] in database [Database_24_Stage] (6). The OS file handle is 0x0000000000000C18. The offset of the latest long I/O is: 0x0000038542bc00 2012-10-04 16:37:27.54 spid8s SQL Server has encountered 13 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\log\Database_24_Stage_log.ldf] in database [Database_24_Stage] (6). The OS file handle is 0x0000000000000C18. The offset of the latest long I/O is: 0x000003858eaa00 2012-10-04 16:39:17.73 spid124 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_COUNT', id 0000001E9C80B010, type 2, Task 0x000000205800E558 : 0, waittime 300 seconds, flags 0x1a, owning task 0x00000020FD944928. Continuing to wait. 2012-10-04 16:42:15.65 spid8s SQL Server has encountered 3 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [C:\db\tempDB\1\Database24_100_Log_2.ldf] in database [Database_24] (5). The OS file handle is 0x0000000000001C18. The offset of the latest long I/O is: 0x000001d0d90e00 2012-10-04 16:44:17.75 spid124 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_COUNT', id 0000001E9C80B010, type 2, Task 0x000000205800E558 : 0, waittime 600 seconds, flags 0x1a, owning task 0x00000020FD944928. Continuing to wait. 2012-10-04 16:48:04.40 spid8s SQL Server has encountered 13 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [C:\db\tempDB\1\Database24_100_Log_2.ldf] in database [Database_24] (5). The OS file handle is 0x0000000000001C18. The offset of the latest long I/O is: 0x000001d10f6e00 2012-10-04 16:49:17.77 spid124 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_COUNT', id 0000001E9C80B010, type 2, Task 0x000000205800E558 : 0, waittime 900 seconds, flags 0x1a, owning task 0x00000020FD944928. Continuing to wait. 2012-10-04 16:49:55.60 spid8s SQL Server has encountered 4 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\log\Database_24_Stage_log.ldf] in database [Database_24_Stage] (6). The OS file handle is 0x0000000000000C18. The offset of the latest long I/O is: 0x00000385d4d400 2012-10-04 16:50:51.19 spid8s SQL Server has encountered 90 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\tempdb\1\tempdev.mdf] in database [tempdb] (2). The OS file handle is 0x0000000000000D18. The offset of the latest long I/O is: 0x00000000226000 2012-10-04 16:53:48.10 spid8s SQL Server has encountered 6 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [C:\db\tempDB\1\Database24_100_Log_2.ldf] in database [Database_24] (5). The OS file handle is 0x0000000000001C18. The offset of the latest long I/O is: 0x000001d13d5e00 2012-10-04 16:55:39.30 spid8s SQL Server has encountered 13 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\log\Database_24_Stage_log.ldf] in database [Database_24_Stage] (6). The OS file handle is 0x0000000000000C18. The offset of the latest long I/O is: 0x000003860b2200 2012-10-04 16:57:36.18 spid53 A time-out occurred while waiting for buffer latch -- type 4, bp 000000211D81E7C0, page 1:303, stat 0x40d, database id: 2, allocation unit Id: 72057595871166464, task 0x0000002056BC5C38 : 0, waittime 300 seconds, flags 0x1a, owning task 0x0000002056BC5C38. Not continuing to wait. 2012-10-04 16:57:55.78 spid8s SQL Server has encountered 2 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [c:\db\tempdb\1\tempdev.mdf] in database [tempdb] (2). The OS file handle is 0x0000000000000D18. The offset of the latest long I/O is: 0x00000000260000 2012-10-04 16:59:31.82 spid8s SQL Server has encountered 6 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [C:\db\tempDB\1\Database24_100_Log_2.ldf] in database [Database_24] (5). The OS file handle is 0x0000000000001C18. The offset of the latest long I/O is: 0x000001d164a800 2012-10-04 17:00:21.98 spid74 Timeout occurred while waiting for latch: class 'ACCESS_METHODS_HOBT_COUNT', id 0000002079F738F0, type 2, Task 0x00000020FD065498 : 0, waittime 300 seconds, flags 0x1a, owning task 0x00000020FD944928. Continuing to wait.