Optimizer memory, memoryclerk_sqloptimizer memory very high and climbing

I have a SQL Server 2012 instance running higher and higher amounts of memory for the memoryclerk_sqloptimizer. We have logging for optimizer memory and it is at 13 GB and climbing, but we have seen it higher and previously had to restart to clear it up. 

Version info: 

Microsoft SQL Server 2012 (SP1) - 11.0.3000.0 (X64) Enterprise Edition (64-bit) on Windows NT 6.1 <X64> (Build 7601: Service Pack 1)

A few queries I ran to debug this: 

SELECT sum(mo.pages_in_bytes)/1024./1024./1024., mo.type, mc.type
FROM sys.dm_os_memory_objects mo
inner join sys.dm_os_memory_clerks mc on mo.page_allocator_address = mc.page_allocator_address
GROUP BY mo.type, mc.type, mc.type
ORDER BY 1 DESC;

select type, name, sum(pages_kb)
from sys.dm_os_memory_clerks
where pages_kb != 0
group by type, name 
order by 3 desc  

The first one has a top sum of 13GB and it is for the memory object type of "MEMOBJ_EXECCOMPILETEMP" and memory clerk "MEMORYCLERK_SQLOPTIMIZER" 

The second query has a top clerk of "MEMORYCLERK_SQLBUFFERPOOL" which is to be expected, and then "MEMORYCLERK_SQLOPTIMIZER". These are at 22768808 and 13747688 for the sum of pages_kb, respectively. 

I have turned on optimize for ad hoc workloads and that had no effect. Of course restarting the server gets us back to normal but eventually, after a week maybe, the optimizer memory gets back to up 10+GB and climbs until there is no memory left for anything else. We do have lots of ad hoc queries but this is nothing new --- we have had lots of ad hoc queries for a very long time and this optimizer memory issue is recent (past few weeks to a month or so ago). 

Comparing to a "normal" time period, our stats show optimizer memory at 0.002 GB while now it is much higher and climbing. Also, probably a victim of memory pressure, latch waits / sec are much higher, but also I noticed workfiles creates /sec and worktables created /sec are higher (300 more on avg and 100 more on avg, respectively). SQL compilations are also higher now. We noticed that when the optimizer memory gets very high, like in to 30GB+ range, even simple queries get compiled on every run, and parse and compile time is the longest time for the query. Again, likely due to less memory available for the buffer pool, we see much more page reads /sec (hard faults) and lower page life expectancy. 

The biggest difference between a "normal" time and now for wait stats is PAGELATCH_UP and PAGEIOLATCH_SH, then CXPACKET, SOS_SCHEDULER_YIELD, and EXECSYNC. The IO related ones are victims of the memory pressure and not a cause. 

Has anyone seen this behavior or does anyone have any ideas on how to get the optimizer to release its memory? We can tell that it gets more memory, releases a little bit, and overtime accumulates many GB of memory, which sets off all of the other high wait stats and counters. 

Thanks!