Opened 4 years ago

Closed 4 years ago

#3216 closed defect (fixed)

StopIteration Exception raised in notification FCGI handler

Reported by: matze Assignee: fhd
Priority: P3 Milestone:
Module: Sitescripts Keywords:
Cc: fhd, sporz, sebastian Blocked By:
Blocking: #3209 Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29329851/

Description

While investigating #3209, we stumbled upon the following error raised by the notification.json FCGI backend:

2015/10/21 00:27:35 [error] 30801#0: *502601808 FastCGI sent in stderr: "Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/flup/server/fcgi_base.py", line 558, in run
    protocolStatus, appStatus = self.server.handler(self)
  File "/usr/lib/pymodules/python2.7/flup/server/fcgi_base.py", line 1118, in handler
    result = self.application(environ, start_response)
  File "/opt/sitescripts/sitescripts/web.py", line 111, in multiplex
    try:
  File "/opt/sitescripts/sitescripts/notifications/web/notification.py", line 108, in notification
    response = _create_response(notifications, groups)
  File "/opt/sitescripts/sitescripts/notifications/web/notification.py", line 96, in _create_response
    "notifications": _get_notifications_to_send(notifications, groups)
  File "/opt/sitescripts/sitescripts/notifications/web/notification.py", line 79, in _get_notifications_to_send
    active_variant = _get_active_variant(notifications, groups)
  File "/opt/sitescripts/sitescripts/notifications/web/notification.py", line 62, in _get_active_variant
    notification = next(x for x in notifications if x["id"] == group_id)
StopIteration", client: ***, server: notification.adblockplus.org, request: "GET /notification.json?addonName=adblockplus&addonVersion=2.6.11&application=firefox&applicationVersion=41.0.2&platform=gecko&platformVersion=41.0.2&lastVersion=201509280005-2%2F1&downloadCount=4%2B HTTP/1.1", host: "notification.adblockplus.org"

There are a bunch of those in a day, per server:

$ zgrep StopIteration /var/log/nginx/error.log.1 | wc -l
145979

Interestingly they do not cause the server to report an error, it still reports them with status 200.

Change History (7)

comment:1 Changed 4 years ago by sporz

This error seems to first occur around the time some of the fcgi servers went down, about September 29. Before that date, it seems to not have existed.

comment:2 Changed 4 years ago by sporz

The error persists up to today.
It seems to only affect clients that have "lastVersion=YYYYMMDDHHMM-2%2F[123]", i.e. are in one of the three test-groups.
It occurs in about half of the requests of clients with those lastVersions.

comment:3 Changed 4 years ago by fhd

  • Cc sebastian added
  • Component changed from Unknown to Sitescripts
  • Owner set to fhd

Looks like a sitescripts issue to me, I'll look into it.

comment:4 Changed 4 years ago by fhd

OK, finally got around to actually looking into this. It's definitely a bug in the handler, here's a test that reproduces it:

  def test_inactive_notification_variant_not_returned(self):
    self.load_notifications_mock.return_value = [
      {"id": "a", "inactive": True}
    ]
    result = json.loads(notification.notification({
      "QUERY_STRING": "lastVersion=197001010000-a/1"
    }, lambda *args: None))
    self.assertEqual(len(result["notifications"]), 0)

Shouldn't be too hard to fix, I'll look into that in a bit.

comment:5 Changed 4 years ago by kzar

  • Priority changed from Unknown to P3
  • Ready set

comment:6 Changed 4 years ago by fhd

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

Finally got around to fixing this, patch is under review.

comment:7 Changed 4 years ago by fhd

  • Resolution set to fixed
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.