I've developed a Chrome Extension that cycles through approximately 1500-2000 pages to collect information from a website and push it onto my own server. I use a Chrome extension, as given the requirements it's much easier to configure than making and parsing from the server side.
The extension is used only for internal purposes and we trigger this job using the chrome.alarm API to run at 3:00 am every day. This alarm when triggers pops open a new tab and runs through the 1500-2000 pages. The problem is when we check in the morning we see the SNAP dialogue after the extension has cycled through about 1500 pages (approx 3/4). I'm presuming this is due to the memory demand placed on Chrome to maintain such an unusually large history?
My question is, what would be the best way to mitigate this? Presumably killing the tab and reopening (after x number of pages) would work but that would slow down the feed and require quite a bit of code re-factoring. Is there any way you can force Chrome to dump the history and would doing so free memory in the immediate session?
Just to add some context, I am running this on a extra small VM with only 1 GB of memory. I appreciate I could upgrade the VM but that really just defers the problem.