Opened on 07/31/2018 at 05:09:47 PM
Closed on 09/21/2018 at 04:17:36 PM
Last modified on 02/07/2019 at 03:23:28 AM
#6822 closed defect (fixed)
adblockpluscore browser test failures are hidden/obscured if unit tests pass
| Reported by: | hfiguiere | Assignee: | hfiguiere |
|---|---|---|---|
| Priority: | P2 | Milestone: | |
| Module: | Core | Keywords: | |
| Cc: | kzar, sebastian, mjethani, tlucas | Blocked By: | |
| Blocking: | Platform: | Unknown / Cross platform | |
| Ready: | yes | Confidential: | no |
| Tester: | Unknown | Verified working: | no |
| Review URL(s): | |||
Description (last modified by kzar)
Environment
adblockpluscore has both unit tests, "browser tests" are more like integration tests which test things like element hiding which cannot easily be tested without a full (albeit headless) browser.
Currently the browser tests run first, then the unit tests. The output isn't combined so you see the individual browser test results, overall browser test result, individual unit test results and finally overall unit test result printed to the screen. Since the output scrolls past very quickly, it can be easy to miss failures in the browser tests therefore, especially if the unit tests pass.
Even worse, since the final message is something like "OK: 2024 assertions (3239ms)" if the unit tests pass, CI systems such as Travis consider the build a success - even if the browser tests failed!
What to change
Combine the output of the unit and browser tests somehow, so that the overall "OK: X assertions (Xms)" message is shared to include both types of tests. If a single browser or unit test fails, an overall failure message should be displayed.
Notes
- With #6391 this problem is exacerbated since the browser tests are run multiple times under different browsers.
Attachments (0)
Change History (9)
comment:1 Changed on 07/31/2018 at 06:42:16 PM by hfiguiere
- Owner set to hfiguiere
comment:2 Changed on 08/03/2018 at 02:50:23 PM by kzar
- Cc mjethani added
- Description modified (diff)
- Priority changed from Unknown to P2
- Ready set
- Summary changed from adblockpluscore test in the browser don't propagate failures to adblockpluscore browser test failures are hidden/obscured if unit tests pass
comment:3 Changed on 08/03/2018 at 02:50:57 PM by kzar
comment:4 Changed on 08/03/2018 at 03:11:00 PM by kzar
- Cc tlucas added
comment:6 Changed on 09/04/2018 at 07:08:08 PM by hfiguiere
- Review URL(s) modified (diff)
- Status changed from new to reviewing
comment:7 Changed on 09/21/2018 at 04:16:32 PM by abpbot
A commit referencing this issue has landed:
Issue 6822 - Report browser test errors
comment:8 Changed on 09/21/2018 at 04:17:36 PM by hfiguiere
- Resolution set to fixed
- Status changed from reviewing to closed
comment:9 Changed on 02/07/2019 at 03:23:28 AM by abpbot
A commit referencing this issue has landed:
Issue 6822 - Report browser test errors
Thanks for filing this Hubert.