Opened 5 years ago

Closed 5 years ago

#1569 closed defect (fixed)

Issue report digest update is inefficient

Reported by: trev Assignee: trev
Priority: P1 Milestone:
Module: Infrastructure Keywords:
Cc: Blocked By:
Blocking: Platform: Unknown
Ready: yes Confidential: no
Tester: Verified working: no
Review URL(s):

http://codereview.adblockplus.org/5694484511719424

Description

Background

When updating issue report digests, we currently need to process 160k issue reports. In order to save memory we process 1000 issue reports at a time. As it turns out, this produced significant stress in the database which had to process 160 rather large queries - the script spends most of the time waiting for the database to retrieve data. Even worse, the memory use of the script is very high regardless as Python garbage collection doesn't seem to be too efficient.

What to change

Process issue reports in steps of 10000, this will allow the script to finish faster and consequently release memory faster.

Change History (2)

comment:1 Changed 5 years ago by trev

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:2 Changed 5 years ago by trev

  • Resolution set to fixed
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.