Opened on 11/13/2014 at 08:12:39 PM
Closed on 11/19/2014 at 02:22:47 PM
#1569 closed defect (fixed)
Issue report digest update is inefficient
Reported by: | trev | Assignee: | trev |
---|---|---|---|
Priority: | P1 | Milestone: | |
Module: | Infrastructure | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Platform: | Unknown | |
Ready: | yes | Confidential: | no |
Tester: | Verified working: | no | |
Review URL(s): |
Description
Background
When updating issue report digests, we currently need to process 160k issue reports. In order to save memory we process 1000 issue reports at a time. As it turns out, this produced significant stress in the database which had to process 160 rather large queries - the script spends most of the time waiting for the database to retrieve data. Even worse, the memory use of the script is very high regardless as Python garbage collection doesn't seem to be too efficient.
What to change
Process issue reports in steps of 10000, this will allow the script to finish faster and consequently release memory faster.
Fixed: https://hg.adblockplus.org/sitescripts/rev/195d8fb7927c