Opened on 05/06/2016 at 10:52:37 PM
Closed on 10/11/2016 at 07:57:52 PM
#4023 closed change (fixed)
Fix "Subscriptions storage full" message - move storage of subscription lists to IndexedDB
| Reported by: | scottlow | Assignee: | oleksandr |
|---|---|---|---|
| Priority: | Unknown | Milestone: | Adblock-Plus-0.9.8-for-Microsoft-Edge |
| Module: | Platform | Keywords: | |
| Cc: | sebastian, oleksandr, scottlow, sporz | Blocked By: | |
| Blocking: | Platform: | Edge | |
| Ready: | yes | Confidential: | no |
| Tester: | Unknown | Verified working: | no |
| Review URL(s): |
https://codereview.adblockplus.org/29348062/ |
||
Description (last modified by sebastian)
Edge has a limit of 5Mb storage in storage.local API and also a limit of 1Mb per storage item. Previously we tried to overcome these limitations by splitting the subscription lists into 1Mb chunks. However turned out that is not enough. Since Edge uses UTF-16 to store data internally our subscriptions list use way more data than the current maximum 5Mb. We then tried to compress the subscription lists before storing them and uncompress them when loading. It did not work out well enough too, because that required a lot of computing, which made the UI highly unresponsive, and anything that was touching the subscription lists took way too much time.
What to change
Instead of trying to fit the data into a limited storage.local API (where the limit is 5Mb), or localStorage API (where the limit is 10Mb) use IndexedDB API. The storage limit there is from 50Mb to 4% of the volume size in most cases, which should be enough for all of our uncompressed storage needs.
Hints for testers
- Make sure that when adding some addional subscriptions, custom filters and whitelisted domains, these are preserved over browser restarts.
- Make sure that when upgrading from the previous Adblock Plus 0.6.9 all filter preferences are preserved.
Attachments (0)
Change History (19)
comment:1 Changed on 05/07/2016 at 01:35:31 AM by sebastian
- Cc sebastian oleksandr added
comment:2 Changed on 05/08/2016 at 07:39:12 AM by oleksandr
- Description modified (diff)
- Keywords edge added
- Owner set to oleksandr
comment:3 Changed on 05/09/2016 at 12:38:12 PM by oleksandr
- Cc scottlow added
comment:4 Changed on 05/09/2016 at 01:05:12 PM by oleksandr
I've managed to reproduce it now on Windows build 14328.
comment:5 Changed on 05/23/2016 at 10:36:25 AM by sebastian
- Keywords edge removed
- Platform changed from Internet Explorer to Edge
comment:6 Changed on 06/10/2016 at 11:27:49 PM by sebastian
- Sensitive unset
comment:7 Changed on 07/20/2016 at 11:58:17 AM by oleksandr
- Review URL(s) modified (diff)
Adding a codereview that has a working 1Mb chunking mechanism. However since there is also a limit of total 5Mb storage per extension it looks like we will have to move away from the chrome.storage.local in Edge. Unless there will be support for unlimitedStorage permission soon.
comment:8 Changed on 07/24/2016 at 10:57:21 AM by revanmj
I was told on support forum to write a comment about my issue in this ticket so here it goes:
I'm using 14393 Insider build, installed ABP and set it up with my filters and whitelist. Unortunately, after closing Edge and launching it again, settings are back to defaults.
On my main PC settings were eventually saved after multiple tries, but on my tablet ABP is still not able to save its settings and goes back to defaults every time.
My settings were: 7 sites on whitelist and adding official malware and tracking lists, polish filters (http://adblocklist.org/adblock-pxf-polish.txt) and cookies notice list (https://dl.dropboxusercontent.com/u/25725152/cookiesblock.txt).
comment:9 Changed on 08/09/2016 at 09:22:47 AM by sporz
- Cc sporz added
comment:10 Changed on 08/20/2016 at 01:44:26 AM by oleksandr
- Component changed from Unknown to Platform
- Description modified (diff)
- Ready set
- Summary changed from The logic for splitting up storage.local into 1MB chunks does not work to Fix "Subscriptions list full" message - move storage of subscription lists to localStorage
- Type changed from defect to change
comment:11 Changed on 08/20/2016 at 01:45:16 AM by oleksandr
- Summary changed from Fix "Subscriptions list full" message - move storage of subscription lists to localStorage to Fix "Subscriptions storage full" message - move storage of subscription lists to localStorage
comment:12 Changed on 08/22/2016 at 01:57:31 PM by oleksandr
- Review URL(s) modified (diff)
comment:13 Changed on 09/12/2016 at 12:07:23 PM by abpbot
A commit referencing this issue has landed:
Issue 4023 - Move storage of subscription lists to localStorage
comment:14 Changed on 09/13/2016 at 01:47:19 PM by oleksandr
- Resolution set to fixed
- Status changed from new to closed
comment:15 Changed on 09/20/2016 at 07:50:04 AM by sebastian
- Milestone set to Adblock-Plus-0.9.7-for-Microsoft-Edge
comment:16 Changed on 10/05/2016 at 11:56:28 AM by oleksandr
- Description modified (diff)
- Resolution fixed deleted
- Review URL(s) modified (diff)
- Status changed from closed to reopened
- Summary changed from Fix "Subscriptions storage full" message - move storage of subscription lists to localStorage to Fix "Subscriptions storage full" message - move storage of subscription lists to IndexedDB
comment:17 Changed on 10/06/2016 at 12:12:06 PM by sebastian
- Description modified (diff)
comment:18 Changed on 10/11/2016 at 07:33:09 PM by abpbot
A commit referencing this issue has landed:
Issue 4023 - Use localforage (IndexedDB) instead of localStorage
comment:19 Changed on 10/11/2016 at 07:57:52 PM by oleksandr
- Resolution set to fixed
- Status changed from reopened to closed
I cannot reproduce this. Does this happen in the version from the bitbucket? What version of Windows/Edge does this happen on?