Hi All,
We have master table where we have data more than 304.5 Million Records. For Loading this table, ETL is taking more than 6 hours every day in production.
To reduce the Load time thought to go for partitioning of table & implement Switch in,Not switch out (As this is master data we can't go for Archiving so Switch out is not required here)
So just created 3 partitions on this table on Datetime column. Created another table (Table_IN) & this table will be the Target/Destination for ETL. Once Data in Loaded into Table_IN, I am merging last 2 partition & Splitting latest one & switching In latest data.
I observed Merging partition is taking hell of time IF both the Partitions are having data. If Last partition is Empty & 2nd last partition is having data & then try to merge it is happening withing a second but if last & 2nd last Partitions are having data then merge is taking more than 30-50 minutes & during these periods, all the table are getting locked, which all on the Partition function for which we are doing merge.
any solution or best approach to do this.
Thanks Shiven:) If Answer is Helpful, Please Vote