Insert Exec Performance issue

Hi,

I have a procedure which has a code something like 

set @SQL = 'exec [LINKED SERVER].[db_name].dbo.[SP_NAME] @param_1 @param_2'

insert into #temp(...)

exec (@SQL)

When I execute SP without insert statement (without insert into #temp(..)) SP returns in 2 seconds. But, when data is inserted into temp table it takes 7-8 seconds.

Also, when this SP is run first time in a day (or say after 3-4 hours) it takes much longer time(30-40 seconds). 

Only 36 records are returned. 

Thanks for your help. Will be happy to provide any other information requried.