Opened 17 months ago

Closed 15 months ago

Last modified 10 months ago

#6822 closed defect (fixed)

adblockpluscore browser test failures are hidden/obscured if unit tests pass

Reported by: hfiguiere Assignee: hfiguiere
Priority: P2 Milestone:
Module: Core Keywords:
Cc: kzar, sebastian, mjethani, tlucas Blocked By:
Blocking: Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29874649/

Description (last modified by kzar)

Environment

adblockpluscore has both unit tests, "browser tests" are more like integration tests which test things like element hiding which cannot easily be tested without a full (albeit headless) browser.

Currently the browser tests run first, then the unit tests. The output isn't combined so you see the individual browser test results, overall browser test result, individual unit test results and finally overall unit test result printed to the screen. Since the output scrolls past very quickly, it can be easy to miss failures in the browser tests therefore, especially if the unit tests pass.

Even worse, since the final message is something like "OK: 2024 assertions (3239ms)" if the unit tests pass, CI systems such as Travis consider the build a success - even if the browser tests failed!

What to change

Combine the output of the unit and browser tests somehow, so that the overall "OK: X assertions (Xms)" message is shared to include both types of tests. If a single browser or unit test fails, an overall failure message should be displayed.

Notes

  • With #6391 this problem is exacerbated since the browser tests are run multiple times under different browsers.

Change History (9)

comment:1 Changed 17 months ago by hfiguiere

  • Owner set to hfiguiere

comment:2 Changed 17 months ago by kzar

  • Cc mjethani added
  • Description modified (diff)
  • Priority changed from Unknown to P2
  • Ready set
  • Summary changed from adblockpluscore test in the browser don't propagate failures to adblockpluscore browser test failures are hidden/obscured if unit tests pass

comment:3 Changed 17 months ago by kzar

Thanks for filing this Hubert.

comment:4 Changed 17 months ago by kzar

  • Cc tlucas added

comment:5 Changed 17 months ago by kzar

  • Description modified (diff)

comment:6 Changed 16 months ago by hfiguiere

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:7 Changed 15 months ago by abpbot

A commit referencing this issue has landed:
Issue 6822 - Report browser test errors

comment:8 Changed 15 months ago by hfiguiere

  • Resolution set to fixed
  • Status changed from reviewing to closed

comment:9 Changed 10 months ago by abpbot

A commit referencing this issue has landed:
Issue 6822 - Report browser test errors

Note: See TracTickets for help on using tickets.