In the error log, I get:
2013-06-02 00:00:37.78 spid22s This instance of SQL Server has been using a process ID of 1268 since 5/15/2013 4:12:15 AM (local) 5/15/2013 8:12:15 AM (UTC). This is an informational message only; no user action is required.
2013-06-02 00:03:21.45 spid71 Error: 701, Severity: 17, State: 123.
2013-06-02 00:03:21.45 spid71 There is insufficient system memory in resource pool 'internal' to run this query.
2013-06-02 00:12:23.20 spid53 Error: 701, Severity: 17, State: 123.
2013-06-02 00:12:23.20 spid53 There is insufficient system memory in resource pool 'internal' to run this query.
2013-06-02 00:12:23.25 spid59 Error: 701, Severity: 17, State: 123.
2013-06-02 00:12:23.25 spid59 There is insufficient system memory in resource pool 'internal' to run this query.
2013-06-02 00:12:23.26 spid58 Error: 701, Severity: 17, State: 123.
2013-06-02 00:12:23.26 spid58 There is insufficient system memory in resource pool 'internal' to run this query.
2013-06-02 00:12:23.31 spid53 Error: 701, Severity: 17, State: 131.
2013-06-02 00:12:23.31 spid53 There is insufficient system memory in resource pool 'internal' to run this query.
2013-06-02 00:30:10.12 spid75 1 transactions rolled forward in database 'AppFabricMonitoringStoreDb' (9). This is an informational message only. No user action is required.
2013-06-02 00:30:14.12 spid75 0 transactions rolled back in database 'AppFabricMonitoringStoreDb' (9). This is an informational message only. No user action is required.
2013-06-02 00:30:20.78 spid75 Recovery completed for database AppFabricMonitoringStoreDb (database ID 9) in 13 second(s) (analysis 51 ms, redo 2264 ms, undo 2347 ms.) This is an informational message only. No user action is required.
2013-06-02 00:32:23.88 spid75 DBCC CHECKDB (AppFabricMonitoringStoreDb) WITH all_errormsgs, no_infomsgs, data_purity executed by NT AUTHORITY\NETWORK SERVICE found 0 errors and repaired 0 errors. Elapsed time: 0 hours 2 minutes
20 seconds. Internal database snapshot has split point LSN = 0000e67d:00000593:0097 and first LSN = 0000e67d:00000591:0001. This is an informational message only. No user action is required.
2013-06-02 00:32:30.27 spid75 DBCC CHECKDB (AppFabricPersistenceStoreDb) WITH all_errormsgs, no_infomsgs, data_purity executed by NT AUTHORITY\NETWORK SERVICE found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes
0 seconds. Internal database snapshot has split point LSN = 0000111b:000000b7:0001 and first LSN = 0000111b:000000b6:0001. This is an informational message only. No user action is required.
2013-06-02 00:32:31.40 spid75 DBCC CHECKDB (ReportServer$SQL2008) WITH all_errormsgs, no_infomsgs, data_purity executed by NT AUTHORITY\NETWORK SERVICE found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes 0 seconds.
Internal database snapshot has split point LSN = 00000066:0000013f:0001 and first LSN = 00000066:0000013e:0001. This is an informational message only. No user action is required.
2013-06-02 00:32:31.91 spid75 DBCC CHECKDB (ReportServer$SQL2008TempDB) WITH all_errormsgs, no_infomsgs, data_purity executed by NT AUTHORITY\NETWORK SERVICE found 0 errors and repaired 0 errors. Elapsed time: 0 hours 0 minutes
0 seconds. Internal database snapshot has split point LSN = 00000017:00000262:0001 and first LSN = 00000017:00000261:0001. This is an informational message only. No user action is required.
At the time of the error, the system is running a Job Agent Job created by Microsoft AppFabric called Microsoft_ApplicationServer_Monitoring_ImportWfEvents_AppFabricMonitoringStoreDb. It has succeeded in every other invocation except this one at 5:06:30
AM. At the same time, the system runs another job named Microsoft_ApplicationServer_Monitoring_ImportWcfEvents_AppFabricMonitoringStoreDb. Both jobs are scheduled to run every 10 seconds. This job has never failed. This is what I see when viewing the Job Agent
History Log:
Date 6/3/2013 5:06:30 AM
Log Job History (Microsoft_ApplicationServer_Monitoring_ImportWfEvents_AppFabricMonitoringStoreDb)
Step ID 1
Server 1049AS_SQL\SQL2008
Job Name Microsoft_ApplicationServer_Monitoring_ImportWfEvents_AppFabricMonitoringStoreDb
Step Name ASImportWfEvents
Duration 00:00:01
Sql Severity 17
Sql Message ID 701
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0
Message
Executed as user: AS_MonitoringDbJobsAdmin. There is insufficient system memory in resource pool 'internal' to run this query. [SQLSTATE 42000] (Error 701). The step failed.
The Windows Event Log shows no entries around this time the System log, and only these SQL events in the Application log around this time.
Configuration details:
Server: Microsoft Windows NT 6.1 (7601), Windows Server 2008 R2 Standard SP1
Installed RAM: 4.00 GB
This is a VMware VM.
Microsoft SQL Server Standard Edition (64-bit)
10.50.1600.1
SQL_Latin1_General_CP1_CI_AS
Min Memory: 0
Max Memory: 2000 MB
This instance is on the same server as a SQL Server 2005 instance (Min Memory:0, Max Memory: 2000 MB). The 2005 instance was quiet at the time of the error in the 2008 R2 instance. Neither IIS nor any other significant services is running on this server.
This 2008 R2 instance is running SSRS and contains the databases for an AppFabric instance that is running on a different server. The only databases in this instance are the two SSRS databases and the two AppFabric databases, all of which are in simple mode.
The AppFabricMonitoringStoreDb.mdf (E:) is 1.5 GB and its log (F:) is 470 MB. tempdb.mdf (C:) is 250 MB and templog (C:) is 25 MB. All are set to allow unlimited growth. All drives have at least 5 GB free.
I had a similar problem last weekend, when the Max Memory for the 2008 R2 instance was set to 1500 MB. I increased it to 2000 MB as an experiment, but that doesn't seem to have helped. Given that the VM is only allocated 4 GB, the two SQL instances can't each
take 2 GB and leave the OS with nothing, so maybe I just need to allocate more RAM to this VM, but I have 4 other VMs exactly like it (same pair of SQL instances, same jobs) that haven't had any such memory issues and they are more heavily used than this one.
I would appreciate any ideas as to how to track this down and fix the source of this issue.
Thanks,
Mark