Configure the environment to automatically run abpcrawler.

[sergz]

Background

In order to automate some quality related issues we need to run crawler on a periodic basis.

What to change

• prepare a single virtual machine (linux-based). It should have enough space to store gathered information.
• configure to run abpcrawler with some interval
• to be continued...

If it takes longer then convert the current issue into meta issue.

Relevant issues

Ticket	Summary	Component
#3848	Setup file server for AA website screenshots	Office-IT
#2413	[crawler] take screenshots for whitelisted and top sites on recurring basis	Infrastructure

[sergz]

Please find the results below to estimate the hardware requirements.

I have run the crawler with 1000 randomly picked URLs from the commit log of ​easylist. BTW, the number of extracted URLs is a bit more than 73k URL.
Some results:
Number of files in the output folder:

• json - 1000
• xml - 957
• jpg - 954

The size of output folder is 662M.
Starting from some point (> 200 URLs) the avg size of files:

• json - ~50K
• xml - ~170K
• jpg - ~500K

Just for reference std of the file sizes is growing with the number of processed URLs only for xml files, for json and jpg it's pretty constant, although quite big.

Firefox is actually eating a lot of memory. I've observed that with 2GM RAM (and 2GB swap) the memory usage (resident memory) is about 1GB regardless of the number of tabs (I tried with 2, 4, 8, 30), however sometimes it grows up to 1.8 GB and it seems can be even bigger, I guess, it depends on the tab content and GC.

Firefox 45 (release).
Ubuntu 15.10, x86_64, 2GB RAM, 2 Cores.

[trev]

Please note that results for memory usage only apply to your specific configuration. If you give Firefox 1 GB it will likely use less memory, with 4 GB it will likely use more. Garbage collection depends on memory pressure.

[fhd]

IMHO this is very important, setting to P2.