Opened 3 years ago

Closed 3 years ago

#4023 closed change (fixed)

Fix "Subscriptions storage full" message - move storage of subscription lists to IndexedDB

Reported by: scottlow Assignee: oleksandr
Priority: Unknown Milestone: Adblock-Plus-0.9.8-for-Microsoft-Edge
Module: Platform Keywords:
Cc: sebastian, oleksandr, scottlow, sporz Blocked By:
Blocking: Platform: Edge
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29348062/
https://codereview.adblockplus.org/29350042/
https://codereview.adblockplus.org/29355962/

Description (last modified by sebastian)

Edge has a limit of 5Mb storage in storage.local API and also a limit of 1Mb per storage item. Previously we tried to overcome these limitations by splitting the subscription lists into 1Mb chunks. However turned out that is not enough. Since Edge uses UTF-16 to store data internally our subscriptions list use way more data than the current maximum 5Mb. We then tried to compress the subscription lists before storing them and uncompress them when loading. It did not work out well enough too, because that required a lot of computing, which made the UI highly unresponsive, and anything that was touching the subscription lists took way too much time.

What to change

Instead of trying to fit the data into a limited storage.local API (where the limit is 5Mb), or localStorage API (where the limit is 10Mb) use IndexedDB API. The storage limit there is from 50Mb to 4% of the volume size in most cases, which should be enough for all of our uncompressed storage needs.

Hints for testers

  • Make sure that when adding some addional subscriptions, custom filters and whitelisted domains, these are preserved over browser restarts.
  • Make sure that when upgrading from the previous Adblock Plus 0.6.9 all filter preferences are preserved.

Change History (19)

comment:1 Changed 3 years ago by sebastian

  • Cc sebastian oleksandr added

comment:2 Changed 3 years ago by oleksandr

  • Description modified (diff)
  • Keywords edge added
  • Owner set to oleksandr

comment:3 Changed 3 years ago by oleksandr

  • Cc scottlow added

I cannot reproduce this. Does this happen in the version from the bitbucket? What version of Windows/Edge does this happen on?

comment:4 Changed 3 years ago by oleksandr

I've managed to reproduce it now on Windows build 14328.

comment:5 Changed 3 years ago by sebastian

  • Keywords edge removed
  • Platform changed from Internet Explorer to Edge

comment:6 Changed 3 years ago by sebastian

  • Sensitive unset

comment:7 Changed 3 years ago by oleksandr

  • Review URL(s) modified (diff)

Adding a codereview that has a working 1Mb chunking mechanism. However since there is also a limit of total 5Mb storage per extension it looks like we will have to move away from the chrome.storage.local in Edge. Unless there will be support for unlimitedStorage permission soon.

Last edited 3 years ago by oleksandr (previous) (diff)

comment:8 Changed 3 years ago by revanmj

I was told on support forum to write a comment about my issue in this ticket so here it goes:

I'm using 14393 Insider build, installed ABP and set it up with my filters and whitelist. Unortunately, after closing Edge and launching it again, settings are back to defaults.

On my main PC settings were eventually saved after multiple tries, but on my tablet ABP is still not able to save its settings and goes back to defaults every time.

My settings were: 7 sites on whitelist and adding official malware and tracking lists, polish filters (http://adblocklist.org/adblock-pxf-polish.txt) and cookies notice list (https://dl.dropboxusercontent.com/u/25725152/cookiesblock.txt).

comment:9 Changed 3 years ago by sporz

  • Cc sporz added

comment:10 Changed 3 years ago by oleksandr

  • Component changed from Unknown to Platform
  • Description modified (diff)
  • Ready set
  • Summary changed from The logic for splitting up storage.local into 1MB chunks does not work to Fix "Subscriptions list full" message - move storage of subscription lists to localStorage
  • Type changed from defect to change

comment:11 Changed 3 years ago by oleksandr

  • Summary changed from Fix "Subscriptions list full" message - move storage of subscription lists to localStorage to Fix "Subscriptions storage full" message - move storage of subscription lists to localStorage

comment:12 Changed 3 years ago by oleksandr

  • Review URL(s) modified (diff)

comment:13 Changed 3 years ago by abpbot

A commit referencing this issue has landed:
Issue 4023 - Move storage of subscription lists to localStorage

comment:14 Changed 3 years ago by oleksandr

  • Resolution set to fixed
  • Status changed from new to closed

comment:15 Changed 3 years ago by sebastian

  • Milestone set to Adblock-Plus-0.9.7-for-Microsoft-Edge

comment:16 Changed 3 years ago by oleksandr

  • Description modified (diff)
  • Resolution fixed deleted
  • Review URL(s) modified (diff)
  • Status changed from closed to reopened
  • Summary changed from Fix "Subscriptions storage full" message - move storage of subscription lists to localStorage to Fix "Subscriptions storage full" message - move storage of subscription lists to IndexedDB

comment:17 Changed 3 years ago by sebastian

  • Description modified (diff)

comment:18 Changed 3 years ago by abpbot

A commit referencing this issue has landed:
Issue 4023 - Use localforage (IndexedDB) instead of localStorage

comment:19 Changed 3 years ago by oleksandr

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.