Look into Firefox unit test failures

[trev]

Background

Our adblockplustests Firefox extension (​https://adblockplus.org/en/source#unittests) currently consistently reports a bunch of failures. There are a few "allowed request not received by server" (most likely outdated tests no longer triggering a request) and a few pop-up blocking related failures.

What to change

Figure out what those failures are about and fix them.

[trev]

Some test failures have been resolved by #340 and #341. What's left is the UI tests failing (these are not ready for Australis - need to be adapted or removed), some content policies failures (it seems that Firefox no longer does some requests that the tests expect) and an intermittent failure in the filter storage test (hangs up when running all tests but works fine when running on its own).

[philll]

The assigned state will be dropped by #403

[tschuster]

So I started looking at this yesterday. We seem to have quite a few test failures that happen intermittently, which makes them harder track down. I also have problems with running single tests.

The easiest problem are the "Icon position" tests, they are just outdated. These tests were written with the icon in the addon-bar, but now our icon is always in the nav-bar, next to the hamburger button.

"Content policy: Dynamically inserted image button" and "Content policy: Web worker" are just not totally correct tests. We need to wait for the load event of the image and worker, because the frame's onload event (where most of the test code executes) can trigger before them.

Notification handling: Target selection is just really slow.

Filter storage read/write: Saving with backups not sure if that test is just super really slow or broken.

[trev]

Replying to tschuster:

Filter storage read/write: Saving with backups not sure if that test is just super really slow or broken.

I guess you were testing on Linux. There are strange intermittent issues with this test on Linux (error message is thrown), this test works correctly when run individually however. There are no issues on Windows and OS X however.

Note that you can use the Rerun link to run individual tests - or you can use ?test=filterStorage_readwrite to run a single test file only.

[tschuster]

Policy tests are now fixed: ​https://hg.adblockplus.org/adblockplustests/rev/688ad723dac9

[tschuster]

Pushed fix for icon tests: ​https://hg.adblockplus.org/adblockplustests/rev/1323a160ed7a

The tests should now usually pass when using a clean Firefox profile, but I am going to do a few test runs before calling this fixed.

[trev]

Actually, there were some more failures last time I tried. Pop-up blocking tests show issues but that should be #443. Also, I've now seen "saving with backups" test fail on Mac OS X (exception in an async callback, test hangs), previously I've only seen that issue on Linux. This test actually executed correctly when I've rerun it.

[tschuster]

You are right, "saving with backups" just failed, but passed in the run before.

A promise chain failed to handle a rejection. Did you forget to '.catch', or did you forget to 'return'?
See https://developer.mozilla.org/Mozilla/JavaScript_code_modules/Promise.jsm/Promise

Date: Wed Aug 06 2014 18:40:20 GMT+0200 (CEST)
Full Message: Component returned failure code: 0x80520006 (NS_ERROR_FILE_TARGET_DOES_NOT_EXIST) [nsIFile.lastModifiedTime]

I will look into that problem. Luckily all other tests seem to be fine. I haven't seen any problem with pop-up blocking before.

[tschuster]

So from what I can tell "saving with backups" just randomly fails sometimes. However if you keep the tab with test page focused and do nothing else in the browser instance it seems to work most of the time. The failure seems to indicate that cloning the file didn't work for some reason.

[trev]

Mass-closing all bugs in Adblock Plus for Firefox module, the codebase of Adblock Plus 3.0 belongs into Platform and User-Interface modules. Old bugs are unlikely to still apply.