Hello.
last night our production server went pop in spectacular style.
The following is the beginning of the 1000's messages from the SQL error log, the whole log is too large to post (it reads from the bottom up):
09/08/2010 19:45:36,spid311,Unknown,MEMORYBROKER_FOR_RESERVE Allocations = 0 Rate = 0 Target Allocations = 37699 Future Allocations = 59935 Last Notification = STABLE 09/08/2010 19:45:36,spid311,Unknown,MEMORYBROKER_FOR_STEAL Allocations = 215273 Rate = 12 Target Allocations = 198849 Future Allocations = 0 Last Notification = SHRINK 09/08/2010 19:45:36,spid311,Unknown,MEMORYBROKER_FOR_CACHE Allocations = 16218 Rate = 2 Target Allocations = 16824 Future Allocations = 0 Last Notification = STABLE 09/08/2010 19:45:36,spid311,Unknown,Big Gateway Configured Units = 1 Available Units = 1 Acquires = 0 Waiters = 0 Threshold Factor = 8 Threshold = -1 09/08/2010 19:45:36,spid311,Unknown,Medium Gateway Configured Units = 4 Available Units = 4 Acquires = 0 Waiters = 0 Threshold Factor = 12 Threshold = -1 09/08/2010 19:45:36,spid311,Unknown,Small Gateway Configured Units = 16 Available Units = 16 Acquires = 0 Waiters = 0 Threshold Factor = 250000 Threshold = 250000 09/08/2010 19:45:36,spid311,Unknown,Optimization Queue Overall Memory = 2205204480 Target Memory = 1628971008 Last Notification = SHRINK Timeout = 6 Early Termination Factor = 5 09/08/2010 19:45:35,spid311,Unknown,Small Query Memory Objects: Grants=0 Waiting=0 Maximum=1775 Available=1775 Limit=1775 09/08/2010 19:45:35,spid311,Unknown,Query Memory Objects: Next Request=0 Waiting For=0 Cost=0 Timeout=0 Wait Time=0 Last Target=35340 09/08/2010 19:45:35,spid311,Unknown,Query Memory Objects: Grants=0 Waiting=0 Maximum=33573 Available=33573 Limit=33573 09/08/2010 19:45:35,spid311,Unknown,Global Memory Objects: Resource= 823 Locks= 230 SE Schema Mgr= 1667 SQLCache= 154 Replication= 2 ServerGlobal= 28 XPGlobal= 2 Xact= 755 SETLS= 4 DatasetMemObjs= 8 SubpDescPmos= 15 SortTables= 1523 09/08/2010 19:45:35,spid311,Unknown,Process physical/virtual memory pressure: 0/0 System physical memory pressure: 0 09/08/2010 19:45:35,spid311,Unknown,Procedure Cache: TotalProcs=282 TotalPages=3855 InUsePages=333 09/08/2010 19:45:35,spid311,Unknown,Buffer Counts: Committed=335840 Target=335840 Hashed=9053 Internal Reservation=67168 External Reservation=0 Stolen Potential=87504 Min Free=372 Visible=335840 Available Paging File=1114443776 09/08/2010 19:45:35,spid311,Unknown,Buffer Distribution: Stolen=215292 Free=95277 Cached=16219 Database (clean)=8298 Database (dirty)=725 I/O=4<c/> Latched=25 09/08/2010 19:45:35,spid311,Unknown,OBJECTSTORE_LOCK_MANAGER (Total) VM Reserved = 8192 KB VM Committed = 8192 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 1816 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,OBJECTSTORE_SERVICE_BROKER (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 256 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,OBJECTSTORE_SNI_PACKET (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 51752 KB MultiPage Allocator = 48 KB 09/08/2010 19:45:35,spid311,Unknown,OBJECTSTORE_LBSS (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 104 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,USERSTORE_SXC (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 408 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,USERSTORE_OBJPERM (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 3632 KB MultiPage Allocator = 48 KB 09/08/2010 19:45:35,spid311,Unknown,USERSTORE_TOKENPERM (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 7256 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,USERSTORE_DBMETADATA (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 17936 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,USERSTORE_SCHEMAMGR (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 13432 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_SYSTEMROWSET (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 1848 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_EVENTS (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERTO (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERREADONLY (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 32 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERRSB (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERUSERCERTLOOKUP (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERDSH (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERKEK (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_BROKERTBLACS (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 320 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_STACKFRAMES (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 0 KB MultiPage Allocator = 8 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_XMLDBATTRIBUTE (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_XMLDBELEMENT (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_XMLDBTYPE (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_VIEWDEFINITIONS (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_NOTIF (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_TEMPTABLES (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 48 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_XPROC (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 80 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_PHDR (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 928 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_SQLCP (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 9592 KB MultiPage Allocator = 16 KB 09/08/2010 19:45:35,spid311,Unknown,CACHESTORE_OBJCP (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 20496 KB MultiPage Allocator = 16 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLSERVICEBROKERTRANSPORT (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 24 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SOSNODE (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 21728 KB MultiPage Allocator = 8264 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_HOST (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 1647496 KB MultiPage Allocator = 22864 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_BHF (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 96 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLXP (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_FULLTEXT (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SNI (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 3232 KB MultiPage Allocator = 16 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLHTTP (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLSERVICEBROKER (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 176 KB MultiPage Allocator = 192 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLCLR (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLCONNECTIONPOOL (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 17080 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLSTORENG (Total) VM Reserved = 15808 KB VM Committed = 15808 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16000 KB MultiPage Allocator = 13016 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLUTILITIES (Total) VM Reserved = 78824 KB VM Committed = 78824 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 1304 KB MultiPage Allocator = 32 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLOPTIMIZER (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 6616 KB MultiPage Allocator = 72 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLQUERYEXEC (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 16 KB MultiPage Allocator = 0 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLBUFFERPOOL (Total) VM Reserved = 2713696 KB VM Committed = 2713696 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 0 KB MultiPage Allocator = 656 KB 09/08/2010 19:45:35,spid311,Unknown,MEMORYCLERK_SQLGENERAL (Total) VM Reserved = 0 KB VM Committed = 0 KB AWE Allocated = 0 KB SM Reserved = 0 KB SM Committed = 0 KB SinglePage Allocator = 8600 KB MultiPage Allocator = 3168 KB 09/08/2010 19:45:35,spid311,Unknown,Memory node Id = 0 VM Reserved = 2870280 KB VM Committed = 2866688 KB AWE Allocated = 0 KB SinglePage Allocator = 1851992 KB MultiPage Allocator = 48520 KB 09/08/2010 19:45:35,spid311,Unknown,Memory Manager VM Reserved = 2874440 KB VM Committed = 2870696 KB AWE Allocated = 0 KB Reserved Memory = 1024 KB Reserved Memory In Use = 0 KB 09/08/2010 19:45:35,spid311,Unknown,Failed allocate pages: FAIL_PAGE_ALLOCATION 2841
blocks of these messages were repeated for 45 mins until this appeared:
09/08/2010 20:26:20,spid108,Unknown,Exception happened when running extended stored procedure 'xp_readerrorlog' in the library 'xpstar90.dll'. SQL Server is terminating process 108. Exception type: Win32 exception; Exception code: 0xc0000005. 09/08/2010 20:26:20,spid108,Unknown,Error: 18002 Severity: 20 State: 1. 09/08/2010 20:26:20,spid108,Unknown,External dump process return code 0x20000001.<nl/>External dump process returned no errors. 09/08/2010 20:26:16,spid108,Unknown,Stack Signature for the dump is 0x49196044 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Failed load of dbghelp.dll - Not enough storage is available to process this command. 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Version-specific dbghelp.dll is not used 09/08/2010 20:26:16,spid108,Unknown,78132A47 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,781329BB Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010AA7A9 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010AA60C Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010AAB5C Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010AAC3C Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01006C5D Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01006F4D Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01006E27 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,0103D30F Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,0103D647 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010232BB Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010257F0 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,0102522F Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D55B2 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D571D Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D58A6 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010232BB Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,010257F0 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,0102522F Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D55B2 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D571D Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,013D58A6 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,014B6349 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01C23C3C Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,0166684E Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01666340 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,016673A4 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,016679ED Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,01665A04 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,089ABC8E Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,089ABB26 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,089AB827 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,* Short Stack Dump 09/08/2010 20:26:16,spid108,Unknown,* ------------------------------------------------------------------------------- 09/08/2010 20:26:16,spid108,Unknown,* ******************************************************************************* 09/08/2010 20:26:16,spid108,Unknown,* SegSs: 00000023: 09/08/2010 20:26:16,spid108,Unknown,* Esp: 0DF7C230: 00000000 0DF7C4F8 863EE640 00001FD8 00000000 0DF7C4F8 09/08/2010 20:26:16,spid108,Unknown,* EFlags: 00010246: 006C0069 00730065 0053005C 004C0051 004D0058 0020004C 09/08/2010 20:26:16,spid108,Unknown,* SegCs: 0000001B: 09/08/2010 20:26:16,spid108,Unknown,* Ebp: 0DF7C270: 0DF7C4E0 089ABB26 0DF7C294 0DF7C2BC 0DF7C2C4 0DF7C79A 09/08/2010 20:26:16,spid108,Unknown,* Eip: 089AB827: 6608B70F FFFEF981 81661874 74FEFFF9 FC4D8B11 E068106A 09/08/2010 20:26:16,spid108,Unknown,* Edx: 0000000A: 09/08/2010 20:26:16,spid108,Unknown,* Ecx: 7C8296F6: 900004C2 90909090 FFFF9090 90B9FFFF 90C27C85 90907C85 09/08/2010 20:26:16,spid108,Unknown,* Ebx: 0DF7C2C4: 00000000 00000000 08CC4238 003A0045 0053005C 004C0051 09/08/2010 20:26:16,spid108,Unknown,* Eax: 00000000: 09/08/2010 20:26:16,spid108,Unknown,* Esi: 0DF7C2BC: 00000000 00000000 00000000 00000000 08CC4238 003A0045 09/08/2010 20:26:16,spid108,Unknown,* Edi: 08CC4238: 089A1680 08CBE708 0DF7C294 000907DA 00080000 00190014 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* MODULE BASE END SIZE 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* EXEC sp_readerrorlog 09/08/2010 20:26:16,spid108,Unknown,* Input Buffer 62 bytes - 09/08/2010 20:26:16,spid108,Unknown,* Access Violation occurred reading address 00000000 09/08/2010 20:26:16,spid108,Unknown,* Exception Code = c0000005 EXCEPTION_ACCESS_VIOLATION 09/08/2010 20:26:16,spid108,Unknown,* Exception Address = 089AB827 Module(UNKNOWN+00000000) 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* 09/08/10 20:26:16 spid 108 09/08/2010 20:26:16,spid108,Unknown,* BEGIN STACK DUMP: 09/08/2010 20:26:16,spid108,Unknown,* 09/08/2010 20:26:16,spid108,Unknown,* ******************************************************************************* 09/08/2010 20:26:16,spid108,Unknown,SqlDumpExceptionHandler: Process 108 generated fatal exception c0000005 EXCEPTION_ACCESS_VIOLATION. SQL Server is terminating this process. 09/08/2010 20:26:16,spid108,Unknown,***Stack Dump being sent to C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG\SQLDump0007.txt 09/08/2010 20:26:16,spid108,Unknown,***Symbol load failed - Proceed to dump without any symbols 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Failed load of dbghelp.dll - Not enough storage is available to process this command. 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Version-specific dbghelp.dll is not used 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Failed load of dbghelp.dll - Not enough storage is available to process this command. 09/08/2010 20:26:16,spid108,Unknown,CImageHelper::Init () Version-specific dbghelp.dll is not used
blocks of the two above messages were repeated until eventually the server shutdown here:
09/08/2010 23:22:47,spid108,Unknown,SQL Trace was stopped due to server shutdown. Trace ID = '1'. This is an informational message only; no user action is required. 09/08/2010 23:22:47,spid108,Unknown,SQL Server must shut down in order to recover a database (database ID 2). The database is either a user database that could not be shut down or a system database. Restart SQL Server. If the database fails to recover after another startup repair or restore the database. 09/08/2010 23:22:47,spid108,Unknown,Error: 3449 Severity: 21 State: 1. 09/08/2010 23:22:47,spid108,Unknown,During undoing of a logged operation in database 'tempdb' an error occurred at log record ID (1807:45976:531). Typically the specific failure is logged previously as an error in the Windows Event Log service. Restore the database or file from a backup or repair the database. 09/08/2010 23:22:47,spid108,Unknown,Error: 3314 Severity: 21 State: 4. 09/08/2010 23:22:47,spid108,Unknown,There is insufficient system memory to run this query. 09/08/2010 23:22:47,spid108,Unknown,Error: 701 Severity: 17 State: 74. 09/08/2010 23:22:47,spid108,Unknown,There is insufficient system memory to run this query. 09/08/2010 23:22:47,spid108,Unknown,Error: 701 Severity: 17 State: 84.
which indicates a problem with tempdb.
At the time the errors started our nightly full backups had just begun, however we have been running them for years at this time and this has never happened before.
After the backups finish we run an number of SSIS jobs to and I think these will be the contributing factor as to why the server went down eventually as it would have become overloaded with requests the it couldn't process. I can accept that but I'm trying to find out why the FAIL_PAGE_ALLOCATION error appeared at a time when I wouldn't have expected to.
There are a few KB articles ( 942661, 2000380 and 2001221 ) on the subject but all refer to batch processing and stored procedures etc. I don't believe any were running at the time but I wasn't there at the time so I couldn't say for sure, this is when it really be nice to look back and see what was running :-)
Version is Standard Edn 9.0.4053 (SP3)
The server started backup ok but any help much would be much appreciated as I could do without this happening again.
Thanks