external/uBlock
1
0
Fork 0
mirror of https://github.com/gorhill/uBlock.git synced 2024-11-10 09:07:54 +01:00
Code Issues Projects Releases Packages Wiki Activity

Expand bidi-trie usage in static network filtering engine

Browse source 
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/761
- https://github.com/uBlockOrigin/uBlock-issues/issues/528

The previous bidi-trie code could only hold filters which
are plain pattern, i.e. no wildcard characters, and which
had no origin option (`domain=`), right and/or left anchor,
and no `csp=` option.

Example of filters that could be moved into a bidi-trie
data structure:

    &ad_box_
    /w/d/capu.php?z=$script,third-party
    ||liveonlinetv247.com/images/muvixx-150x50-watch-now-in-hd-play-btn.gif

Examples of filters that could NOT be moved to a bidi-trie:

    -adap.$domain=~l-adap.org
    /tsc.php?*&ses=
    ||ibsrv.net/*forumsponsor$domain=[...]
    @@||imgspice.com/jquery.cookie.js|$script
    ||view.atdmt.com^*/iview/$third-party
    ||postimg.cc/image/$csp=[...]

Ideally the filters above should be able to be moved to a
bidi-trie since they are basically plain patterns, or at
least partially moved to a bidi-trie when there is only a
single wildcard (i.e. made of two plain patterns).

Also, there were two distinct bidi-tries in which
plain-pattern filters can be moved to: one for patterns
without hostname anchoring and another one for patterns
with hostname-anchoring. This was required because the
hostname-anchored patterns have an extra condition which
is outside the bidi-trie knowledge.

This commit expands the number of filters which can be
stored in the bidi-trie, and also remove the need to
use two distinct bidi-tries.

- Added ability to associate a pattern with an integer
  in the bidi-trie [1].
    - The bidi-trie match code passes this externally
      provided integer when calling an externally
      provided method used for testing extra conditions
      that may be present for a plain pattern found to
      be matching in the bidi-trie.

- Decomposed existing filters into smaller logical units:
    - FilterPlainLeftAnchored =>
        FilterPatternPlain +
        FilterAnchorLeft
    - FilterPlainRightAnchored =>
        FilterPatternPlain +
        FilterAnchorRight
    - FilterExactMatch =>
        FilterPatternPlain +
        FilterAnchorLeft +
        FilterAnchorRight
    - FilterPlainHnAnchored =>
        FilterPatternPlain +
        FilterAnchorHn
    - FilterWildcard1 =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ]
    - FilterWildcard1HnAnchored =>
        FilterPatternPlain + [
          FilterPatternLeft or
          FilterPatternRight
        ] +
        FilterAnchorHn
    - FilterGenericHnAnchored =>
        FilterPatternGeneric +
        FilterAnchorHn
    - FilterGenericHnAndRightAnchored =>
        FilterPatternGeneric +
        FilterAnchorRight +
        FilterAnchorHn
    - FilterOriginMixedSet =>
        FilterOriginMissSet +
        FilterOriginHitSet
    - Instances of FilterOrigin[...], FilterDataHolder
      can also be added to a composite filter to
      represent `domain=` and `csp=` options.

- Added a new filter class, FilterComposite, for
  filters which are a combination of two or more
  logical units. A FilterComposite instance is a
  match when *all* filters composing it are a
  match.

Since filters are now encoded into combination of
smaller units, it becomes possible to extract the
FilterPatternPlain component and store it in the
bidi-trie, and use the integer as a handle for the
remaining extra conditions, if any.

Since a single pattern in the bidi-trie may be a
component for different filters, the associated
integer points to a sequence of extra conditions,
and a match occurs as soon as one of the extra
conditions (which may itself be a sequence of
conditions) is fulfilled.

Decomposing filters which are currently single
instance into sequences of smaller logical filters
means increasing the storage and CPU overhead when
evaluating such filters. The CPU overhead is
compensated by the fact that more filters can now
moved into the bidi-trie, where the first match is
efficiently evaluated. The extra conditions have to
be evaluated if and only if there is a match in the
bidi-trie.

The storage overhead is compensated by the
bidi-trie's intrinsic nature of merging similar
patterns.

Furthermore, the storage overhead is reduced by no
longer using JavaScript array to store collection
of filters (which is what FilterComposite is):
the same technique used in [2] is imported to store
sequences of filters.

A sequence of filters is a sequence of integer pairs
where the first integer is an index to an actual
filter instance stored in a global array of filters
(`filterUnits`), while the second integer is an index
to the next pair in the sequence -- which means all
sequences of filters are encoded in one single array
of integers (`filterSequences` => Uint32Array). As
a result, a sequence of filters can be represented by
one single integer -- an index to the first pair --
regardless of the number of filters in the sequence.

This representation is further leveraged to replace
the use of JavaScript array in FilterBucket [3],
which used a JavaScript array to store collection
of filters. Doing so means there is no more need for
FilterPair [4], which purpose was to be a lightweight
representation when there was only two filters in a
collection.

As a result of the above changes, the map of `token`
(integer)  => filter instance (object) used to
associate tokens to filters or collections of filters
is replaced with a more efficient map of `token`
(integer) to filter unit index (integer) to lookup a
filter object from the global `filterUnits` array.

Another consequence of using one single global
array to store all filter instances means we can reuse
existing instances when a logical filter instance is
parameter-less, which is the case for FilterAnchorLeft,
FilterAnchorRight, FilterAnchorHn, the index to these
single instances is reused where needed.

`urlTokenizer` now stores the character codes of the
scanned URL into a bidi-trie buffer, for reuse when
string matching methods are called.

New method: `tokenHistogram()`, used to generate
histograms of occurrences of token extracted from URLs
in built-in benchmark. The top results of the "miss"
histogram are used as "bad tokens", i.e. tokens to
avoid if possible when compiling filter lists.

All plain pattern strings are now stored in the
bidi-trie memory buffer, regardless of whether they
will be used in the trie proper or not.

Three methods have been added to the bidi-trie to test
stored string against the URL which is also stored in
then bidi-trie.

FilterParser is now instanciated on demand and
released when no longer used.

***

[1] 135a45a878/src/js/strie.js (L120)
[2] e94024d350
[3] 135a45a878/src/js/static-net-filtering.js (L1630)
[4] 135a45a878/src/js/static-net-filtering.js (L1566)
This commit is contained in:
Raymond Hill 2019-10-21 08:15:58 -04:00
parent 9f069850e5
commit 7971b22385
No known key found for this signature in database
GPG key ID: 25E1490B761470C2
12 changed files with 2540 additions and 2094 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
src/js/background.js
View file

@ -141,8 +141,8 @@ const µBlock = (( ) => { // jshint ignore:line
// Read-only
systemSettings: {
compiledMagic: 21, // Increase when compiled format changes
selfieMagic: 22, // Increase when selfie format changes
compiledMagic: 23, // Increase when compiled format changes
selfieMagic: 23, // Increase when selfie format changes
},
restoreBackupSettings: {

1
src/js/document-blocked.js
View file

@ -44,7 +44,6 @@ let details = {};
(async ( ) => {
const response = await messaging.send('documentBlocked', {
what: 'listsFromNetFilter',
compiledFilter: details.fc,
rawFilter: details.fs,
});
if ( response instanceof Object === false ) { return; }

2
src/js/hntrie.js
View file

@ -560,6 +560,8 @@ HNTrieContainer.prototype.HNTrieRef = class {
this.container = container;
this.iroot = iroot;
this.size = size;
this.needle = '';
this.last = -1;
}
add(hn) {

3
src/js/logger-ui.js
View file

@ -659,7 +659,6 @@ const viewPort = (( ) => {
}
if ( filteringType === 'static' ) {
divcl.add('canLookup');
div.setAttribute('data-filter', filter.compiled);
} else if ( filteringType === 'cosmetic' ) {
divcl.add('canLookup');
divcl.toggle('isException', filter.raw.startsWith('#@#'));
@ -1465,7 +1464,6 @@ const reloadTab = function(ev) {
const fillSummaryPaneFilterList = async function(rows) {
const rawFilter = targetRow.children[1].textContent;
const compiledFilter = targetRow.getAttribute('data-filter');
const nodeFromFilter = function(filter, lists) {
const fragment = document.createDocumentFragment();
@ -1524,7 +1522,6 @@ const reloadTab = function(ev) {
if ( targetRow.classList.contains('networkRealm') ) {
const response = await messaging.send('loggerUI', {
what: 'listsFromNetFilter',
compiledFilter: compiledFilter,
rawFilter: rawFilter,
});
handleResponse(response);

14
src/js/messaging.js
View file

@ -66,14 +66,18 @@ const onMessage = function(request, sender, callback) {
case 'listsFromNetFilter':
µb.staticFilteringReverseLookup.fromNetFilter(
request.compiledFilter,
request.rawFilter,
callback
);
request.rawFilter
).then(response => {
callback(response);
});
return;
case 'listsFromCosmeticFilter':
µb.staticFilteringReverseLookup.fromCosmeticFilter(request, callback);
µb.staticFilteringReverseLookup.fromCosmeticFilter(
request
).then(response => {
callback(response);
});
return;
case 'reloadAllFilters':

3
src/js/redirect-engine.js
View file

@ -737,6 +737,9 @@ RedirectEngine.prototype.loadBuiltinResources = function() {
store(name, reader.result);
resolve();
};
reader.onabort = reader.onerror = ( ) => {
resolve();
};
reader.readAsDataURL(blob);
});
};

44
src/js/reverselookup.js
View file

@ -39,9 +39,9 @@ let messageId = 1;
const onWorkerMessage = function(e) {
const msg = e.data;
const callback = pendingResponses.get(msg.id);
const resolver = pendingResponses.get(msg.id);
pendingResponses.delete(msg.id);
callback(msg.response);
resolver(msg.response);
};
/******************************************************************************/
@ -55,6 +55,9 @@ const stopWorker = function() {
worker.terminate();
worker = null;
needLists = true;
for ( const resolver of pendingResponses.values() ) {
resolver();
}
pendingResponses.clear();
};
@ -127,36 +130,34 @@ const initWorker = function() {
/******************************************************************************/
const fromNetFilter = async function(compiledFilter, rawFilter, callback) {
if ( typeof callback !== 'function' ) {
return;
}
const fromNetFilter = async function(rawFilter) {
if ( typeof rawFilter !== 'string' || rawFilter === '' ) { return; }
if ( compiledFilter === '' || rawFilter === '' ) {
callback();
const µb = µBlock;
const writer = new µb.CompiledLineIO.Writer();
if ( µb.staticNetFilteringEngine.compile(rawFilter, writer) === false ) {
return;
}
await initWorker();
const id = messageId++;
const message = {
worker.postMessage({
what: 'fromNetFilter',
id: id,
compiledFilter: compiledFilter,
compiledFilter: writer.last(),
rawFilter: rawFilter
};
pendingResponses.set(id, callback);
worker.postMessage(message);
});
return new Promise(resolve => {
pendingResponses.set(id, resolve);
});
};
/******************************************************************************/
const fromCosmeticFilter = async function(details, callback) {
if ( typeof callback !== 'function' ) { return; }
if ( details.rawFilter === '' ) {
callback();
const fromCosmeticFilter = async function(details) {
if ( typeof details.rawFilter !== 'string' || details.rawFilter === '' ) {
return;
}
@ -164,7 +165,7 @@ const fromCosmeticFilter = async function(details, callback) {
const id = messageId++;
const hostname = µBlock.URI.hostnameFromURI(details.url);
pendingResponses.set(id, callback);
worker.postMessage({
what: 'fromCosmeticFilter',
id: id,
@ -182,6 +183,11 @@ const fromCosmeticFilter = async function(details, callback) {
) === 2,
rawFilter: details.rawFilter
});
return new Promise(resolve => {
pendingResponses.set(id, resolve);
});
};
/******************************************************************************/

3998
src/js/static-net-filtering.js
View file

File diff suppressed because it is too large Load diff

1
src/js/storage.js
View file

@ -1101,6 +1101,7 @@
catch (reason) {
log.info(reason);
}
destroy();
return false;
};

504
src/js/strie.js
View file

@ -31,12 +31,13 @@
A BidiTrieContainer is mostly a large buffer in which distinct but related
tries are stored. The memory layout of the buffer is as follow:
0-255: reserved
256-259: offset to start of trie data section (=> trie0)
260-263: offset to end of trie data section (=> trie1)
264-267: offset to start of character data section (=> char0)
268-271: offset to end of character data section (=> char1)
272: start of trie data section
0-2047: haystack section
2048-2051: number of significant characters in the haystack
2052-2055: offset to start of trie data section (=> trie0)
2056-2059: offset to end of trie data section (=> trie1)
2060-2063: offset to start of character data section (=> char0)
2064-2067: offset to end of character data section (=> char1)
2068: start of trie data section
+--------------+
Normal cell: | And | If "Segment info" matches:
@ -99,35 +100,56 @@
*/
const PAGE_SIZE = 65536;
// i32 / i8
const TRIE0_SLOT = 256 >>> 2; // 64 / 256
const TRIE1_SLOT = TRIE0_SLOT + 1; // 65 / 260
const CHAR0_SLOT = TRIE0_SLOT + 2; // 66 / 264
const CHAR1_SLOT = TRIE0_SLOT + 3; // 67 / 268
const TRIE0_START = TRIE0_SLOT + 4 << 2; // 272
const PAGE_SIZE = 65536*2;
// i32 / i8
const HAYSTACK_START = 0;
const HAYSTACK_SIZE = 2048;
const HAYSTACK_SIZE_SLOT = HAYSTACK_SIZE >>> 2; // 512 / 2048
const TRIE0_SLOT = HAYSTACK_SIZE_SLOT + 1; // 512 / 2052
const TRIE1_SLOT = HAYSTACK_SIZE_SLOT + 2; // 513 / 2056
const CHAR0_SLOT = HAYSTACK_SIZE_SLOT + 3; // 514 / 2060
const CHAR1_SLOT = HAYSTACK_SIZE_SLOT + 4; // 515 / 2064
const TRIE0_START = HAYSTACK_SIZE_SLOT + 5 << 2; // 2068
// TODO: need a few slots for result values if WASM-ing
const CELL_BYTE_LENGTH = 12;
const MIN_FREE_CELL_BYTE_LENGTH = CELL_BYTE_LENGTH * 4;
const MIN_FREE_CELL_BYTE_LENGTH = CELL_BYTE_LENGTH * 8;
const CELL_AND = 0;
const CELL_OR = 1;
const BCELL_RIGHT_AND = 0;
const BCELL_LEFT_AND = 1;
const SEGMENT_INFO = 2;
const BCELL_NEXT_AND = 0;
const BCELL_ALT_AND = 1;
const BCELL_EXTRA = 2;
const BCELL_EXTRA_MAX = 0x00FFFFFF;
const toSegmentInfo = (aL, l, r) => ((r - l) << 24) | (aL + l);
const roundToPageSize = v => (v + PAGE_SIZE-1) & ~(PAGE_SIZE-1);
µBlock.BidiTrieContainer = class {
constructor(details) {
constructor(details, extraHandler) {
if ( details instanceof Object === false ) { details = {}; }
const len = (details.byteLength || 0) + PAGE_SIZE-1 & ~(PAGE_SIZE-1);
this.buf = new Uint8Array(Math.max(len, 131072));
this.buf32 = new Uint32Array(this.buf.buffer);
const len = roundToPageSize(details.byteLength || 0);
const minInitialLen = PAGE_SIZE * 4;
this.buf8 = new Uint8Array(Math.max(len, minInitialLen));
this.buf32 = new Uint32Array(this.buf8.buffer);
this.buf32[TRIE0_SLOT] = TRIE0_START;
this.buf32[TRIE1_SLOT] = this.buf32[TRIE0_SLOT];
this.buf32[CHAR0_SLOT] = details.char0 || 65536;
this.buf32[CHAR0_SLOT] = details.char0 || (minInitialLen >>> 1);
this.buf32[CHAR1_SLOT] = this.buf32[CHAR0_SLOT];
this.haystack = this.buf8.subarray(
HAYSTACK_START,
HAYSTACK_START + HAYSTACK_SIZE
);
this.haystackSize = 0;
this.extraHandler = extraHandler;
this.textDecoder = null;
this.$l = 0;
this.$r = 0;
this.$iu = 0;
}
//--------------------------------------------------------------------------
@ -139,16 +161,16 @@ const SEGMENT_INFO = 2;
this.buf32[CHAR1_SLOT] = this.buf32[CHAR0_SLOT];
}
matches(iroot, a, i) {
matches(iroot, i) {
const buf32 = this.buf32;
const buf8 = this.buf;
const buf8 = this.buf8;
const char0 = buf32[CHAR0_SLOT];
const aR = a.length;
const aR = this.haystackSize;
let icell = iroot;
let al = i;
let c, v, bl, n;
for (;;) {
c = a.charCodeAt(al);
c = buf8[al];
al += 1;
// find first segment with a first-character match
for (;;) {
@ -156,43 +178,51 @@ const SEGMENT_INFO = 2;
bl = char0 + (v & 0x00FFFFFF);
if ( buf8[bl] === c ) { break; }
icell = buf32[icell+CELL_OR];
if ( icell === 0 ) { return -1; }
if ( icell === 0 ) { return false; }
}
// all characters in segment must match
n = v >>> 24;
if ( n > 1 ) {
n -= 1;
if ( (al + n) > aR ) { return -1; }
if ( (al + n) > aR ) { return false; }
bl += 1;
for ( let i = 0; i < n; i++ ) {
if ( a.charCodeAt(al+i) !== buf8[bl+i] ) { return -1; }
if ( buf8[al+i] !== buf8[bl+i] ) { return false; }
}
al += n;
}
// next segment
icell = buf32[icell+CELL_AND];
if ( /* icell === 0 || */ buf32[icell+SEGMENT_INFO] === 0 ) {
const inext = buf32[icell+BCELL_LEFT_AND];
if ( inext === 0 ) { return (i << 16) | al; }
const r = this.matchesLeft(inext, a, i);
if ( r !== -1 ) { return (r << 16) | al; }
icell = buf32[icell+CELL_AND];
if ( icell === 0 ) { return -1; }
const ix = buf32[icell+BCELL_EXTRA];
if ( ix <= BCELL_EXTRA_MAX ) {
if ( ix !== 0 ) {
const iu = ix === 1 ? -1 : this.extraHandler(i, al, ix);
if ( iu !== 0 ) {
this.$l = i; this.$r = al; this.$iu = iu; return true;
}
}
let inext = buf32[icell+BCELL_ALT_AND];
if ( inext !== 0 ) {
if ( this.matchesLeft(inext, i, al) ) { return true; }
}
inext = buf32[icell+BCELL_NEXT_AND];
if ( inext === 0 ) { return false; }
icell = inext;
}
if ( al === aR ) { return -1; }
if ( al === aR ) { return false; }
}
}
matchesLeft(iroot, a, i) {
matchesLeft(iroot, i, r) {
const buf32 = this.buf32;
const buf8 = this.buf;
const buf8 = this.buf8;
const char0 = buf32[CHAR0_SLOT];
let icell = iroot;
let ar = i;
let c, v, br, n;
for (;;) {
ar -= 1;
c = a.charCodeAt(ar);
c = buf8[ar];
// find first segment with a first-character match
for (;;) {
v = buf32[icell+SEGMENT_INFO];
@ -200,21 +230,31 @@ const SEGMENT_INFO = 2;
br = char0 + (v & 0x00FFFFFF) + n - 1;
if ( buf8[br] === c ) { break; }
icell = buf32[icell+CELL_OR];
if ( icell === 0 ) { return -1; }
if ( icell === 0 ) { return false; }
}
// all characters in segment must match
if ( n > 1 ) {
n -= 1;
if ( n > ar ) { return -1; }
if ( n > ar ) { return false; }
for ( let i = 1; i <= n; i++ ) {
if ( a.charCodeAt(ar-i) !== buf8[br-i] ) { return -1; }
if ( buf8[ar-i] !== buf8[br-i] ) { return false; }
}
ar -= n;
}
// next segment
icell = buf32[icell+CELL_AND];
if ( icell === 0 || buf32[icell+SEGMENT_INFO] === 0 ) { return ar; }
if ( ar === 0 ) { return -1; }
const ix = buf32[icell+BCELL_EXTRA];
if ( ix <= BCELL_EXTRA_MAX ) {
if ( ix !== 0 ) {
const iu = ix === 1 ? -1 : this.extraHandler(ar, r, ix);
if ( iu !== 0 ) {
this.$l = ar; this.$r = r; this.$iu = iu; return true;
}
}
icell = buf32[icell+BCELL_NEXT_AND];
if ( icell === 0 ) { return false; }
}
if ( ar === 0 ) { return false; }
}
}
@ -238,43 +278,47 @@ const SEGMENT_INFO = 2;
return [ trieRef.iroot, trieRef.size ];
}
add(iroot, a, i = 0) {
const aR = a.length;
add(iroot, aL0, n, pivot = 0) {
const aR = n;
if ( aR === 0 ) { return 0; }
// grow buffer if needed
// Grow buffer if needed. The characters are already in our character
// data buffer, so we do not need to grow character data buffer.
if (
(this.buf32[CHAR0_SLOT] - this.buf32[TRIE1_SLOT]) < MIN_FREE_CELL_BYTE_LENGTH ||
(this.buf.length - this.buf32[CHAR1_SLOT]) < 256
(this.buf32[CHAR0_SLOT] - this.buf32[TRIE1_SLOT]) <
MIN_FREE_CELL_BYTE_LENGTH
) {
this.growBuf(MIN_FREE_CELL_BYTE_LENGTH, 256);
this.growBuf(MIN_FREE_CELL_BYTE_LENGTH, 0);
}
const buf32 = this.buf32;
const char0 = buf32[CHAR0_SLOT];
let icell = iroot;
let aL = char0 + aL0;
// special case: first node in trie
if ( buf32[icell+SEGMENT_INFO] === 0 ) {
buf32[icell+SEGMENT_INFO] = this.addSegment(a, i, aR);
return this.addLeft(icell, a, i);
buf32[icell+SEGMENT_INFO] = toSegmentInfo(aL0, pivot, aR);
return this.addLeft(icell, aL0, pivot);
}
const buf8 = this.buf;
const char0 = buf32[CHAR0_SLOT];
let al = i;
const buf8 = this.buf8;
let al = pivot;
let inext;
// find a matching cell: move down
for (;;) {
const binfo = buf32[icell+SEGMENT_INFO];
// length of segment
const bR = binfo >>> 24;
// skip boundary cells
if ( binfo === 0 ) {
icell = buf32[icell+BCELL_RIGHT_AND];
if ( bR === 0 ) {
icell = buf32[icell+BCELL_NEXT_AND];
continue;
}
let bl = char0 + (binfo & 0x00FFFFFF);
// if first character is no match, move to next descendant
if ( buf8[bl] !== a.charCodeAt(al) ) {
if ( buf8[bl] !== buf8[aL+al] ) {
inext = buf32[icell+CELL_OR];
if ( inext === 0 ) {
inext = this.addCell(0, 0, this.addSegment(a, al, aR));
inext = this.addCell(0, 0, toSegmentInfo(aL0, al, aR));
buf32[icell+CELL_OR] = inext;
return this.addLeft(inext, a, i);
return this.addLeft(inext, aL0, pivot);
}
icell = inext;
continue;
@ -283,12 +327,11 @@ const SEGMENT_INFO = 2;
let bi = 1;
al += 1;
// find 1st mismatch in rest of segment
const bR = binfo >>> 24;
if ( bR !== 1 ) {
for (;;) {
if ( bi === bR ) { break; }
if ( al === aR ) { break; }
if ( buf8[bl+bi] !== a.charCodeAt(al) ) { break; }
if ( buf8[bl+bi] !== buf8[aL+al] ) { break; }
bi += 1;
al += 1;
}
@ -297,7 +340,7 @@ const SEGMENT_INFO = 2;
if ( bi === bR ) {
// needle remainder: no
if ( al === aR ) {
return this.addLeft(icell, a, i);
return this.addLeft(icell, aL0, pivot);
}
// needle remainder: yes
inext = buf32[icell+CELL_AND];
@ -306,81 +349,97 @@ const SEGMENT_INFO = 2;
continue;
}
// add needle remainder
icell = this.addCell(0, 0, this.addSegment(a, al, aR));
icell = this.addCell(0, 0, toSegmentInfo(aL0, al, aR));
buf32[inext+CELL_AND] = icell;
return this.addLeft(icell, a, i);
return this.addLeft(icell, aL0, pivot);
}
// some characters matched
// split current segment
bl -= char0;
buf32[icell+SEGMENT_INFO] = bi << 24 | bl;
inext = this.addCell(
buf32[icell+CELL_AND],
0,
bR - bi << 24 | bl + bi
buf32[icell+CELL_AND], 0, bR - bi << 24 | bl + bi
);
buf32[icell+CELL_AND] = inext;
// needle remainder: no = need boundary cell
if ( al === aR ) {
return this.addLeft(icell, a, i);
return this.addLeft(icell, aL0, pivot);
}
// needle remainder: yes = need new cell for remaining characters
icell = this.addCell(0, 0, this.addSegment(a, al, aR));
icell = this.addCell(0, 0, toSegmentInfo(aL0, al, aR));
buf32[inext+CELL_OR] = icell;
return this.addLeft(icell, a, i);
return this.addLeft(icell, aL0, pivot);
}
}
addLeft(icell, a, i) {
addLeft(icell, aL0, pivot) {
const buf32 = this.buf32;
const char0 = buf32[CHAR0_SLOT];
let aL = aL0 + char0;
// fetch boundary cell
let inext = buf32[icell+CELL_AND];
let iboundary = buf32[icell+CELL_AND];
// add boundary cell if none exist
if ( inext === 0 || buf32[inext+SEGMENT_INFO] !== 0 ) {
const iboundary = this.allocateCell();
if (
iboundary === 0 ||
buf32[iboundary+SEGMENT_INFO] > BCELL_EXTRA_MAX
) {
const inext = iboundary;
iboundary = this.allocateCell();
buf32[icell+CELL_AND] = iboundary;
buf32[iboundary+BCELL_RIGHT_AND] = inext;
if ( i === 0 ) { return 1; }
buf32[iboundary+BCELL_LEFT_AND] = this.allocateCell();
inext = iboundary;
buf32[iboundary+BCELL_NEXT_AND] = inext;
if ( pivot === 0 ) { return iboundary; }
}
// shortest match is always first so no point storing whatever is left
if ( buf32[inext+BCELL_LEFT_AND] === 0 ) {
return i === 0 ? 0 : 1;
// shortest match with no extra conditions will always win
if ( buf32[iboundary+BCELL_EXTRA] === 1 ) {
return iboundary;
}
// bail out if no left segment
if ( i === 0 ) {
buf32[inext+BCELL_LEFT_AND] = 0;
return 1;
}
if ( pivot === 0 ) { return iboundary; }
// fetch root cell of left segment
icell = buf32[inext+BCELL_LEFT_AND];
icell = buf32[iboundary+BCELL_ALT_AND];
if ( icell === 0 ) {
icell = this.allocateCell();
buf32[iboundary+BCELL_ALT_AND] = icell;
}
// special case: first node in trie
if ( buf32[icell+SEGMENT_INFO] === 0 ) {
buf32[icell+SEGMENT_INFO] = this.addSegment(a, 0, i);
return 1;
buf32[icell+SEGMENT_INFO] = toSegmentInfo(aL0, 0, pivot);
iboundary = this.allocateCell();
buf32[icell+CELL_AND] = iboundary;
return iboundary;
}
const buf8 = this.buf;
const char0 = buf32[CHAR0_SLOT];
let ar = i;
const buf8 = this.buf8;
let ar = pivot, inext;
// find a matching cell: move down
for (;;) {
const binfo = buf32[icell+SEGMENT_INFO];
// skip boundary cells
if ( binfo === 0 ) {
icell = buf32[icell+CELL_AND];
continue;
if ( binfo <= BCELL_EXTRA_MAX ) {
inext = buf32[icell+CELL_AND];
if ( inext !== 0 ) {
icell = inext;
continue;
}
iboundary = this.allocateCell();
buf32[icell+CELL_AND] =
this.addCell(iboundary, 0, toSegmentInfo(aL0, 0, ar));
// TODO: boundary cell might be last
// add remainder + boundary cell
return iboundary;
}
const bL = char0 + (binfo & 0x00FFFFFF);
const bR = bL + (binfo >>> 24);
let br = bR;
// if first character is no match, move to next descendant
if ( buf8[br-1] !== a.charCodeAt(ar-1) ) {
if ( buf8[br-1] !== buf8[aL+ar-1] ) {
inext = buf32[icell+CELL_OR];
if ( inext === 0 ) {
inext = this.addCell(0, 0, this.addSegment(a, 0, ar));
iboundary = this.allocateCell();
inext = this.addCell(
iboundary, 0, toSegmentInfo(aL0, 0, ar)
);
buf32[icell+CELL_OR] = inext;
return 1;
return iboundary;
}
icell = inext;
continue;
@ -393,37 +452,52 @@ const SEGMENT_INFO = 2;
for (;;) {
if ( br === bL ) { break; }
if ( ar === 0 ) { break; }
if ( buf8[br-1] !== a.charCodeAt(ar-1) ) { break; }
if ( buf8[br-1] !== buf8[aL+ar-1] ) { break; }
br -= 1;
ar -= 1;
}
}
// all segment characters matched
// a: ...vvvvvvv
// b: vvvvvvv
if ( br === bL ) {
inext = buf32[icell+CELL_AND];
// needle remainder: no
// a: vvvvvvv
// b: vvvvvvv
// r: 0 & vvvvvvv
if ( ar === 0 ) {
// boundary cell already present
if ( inext === 0 || buf32[inext+SEGMENT_INFO] === 0 ) {
return 0;
if ( buf32[inext+BCELL_EXTRA] <= BCELL_EXTRA_MAX ) {
return inext;
}
// need boundary cell
buf32[icell+CELL_AND] = this.addCell(inext, 0, 0);
iboundary = this.allocateCell();
buf32[iboundary+CELL_AND] = inext;
buf32[icell+CELL_AND] = iboundary;
return iboundary;
}
// needle remainder: yes
// a: yyyyyyyvvvvvvv
// b: vvvvvvv
else {
if ( inext !== 0 ) {
icell = inext;
continue;
}
// TODO: we should never reach here because there will
// always be a boundary cell.
debugger; // jshint ignore:line
// boundary cell + needle remainder
inext = this.addCell(0, 0, 0);
buf32[icell+CELL_AND] = inext;
buf32[inext+CELL_AND] =
this.addCell(0, 0, this.addSegment(a, 0, ar));
this.addCell(0, 0, toSegmentInfo(aL0, 0, ar));
}
}
// some segment characters matched
// a: ...vvvvvvv
// b: yyyyyyyvvvvvvv
else {
// split current cell
buf32[icell+SEGMENT_INFO] = (bR - br) << 24 | (br - char0);
@ -432,25 +506,38 @@ const SEGMENT_INFO = 2;
0,
(br - bL) << 24 | (bL - char0)
);
buf32[icell+CELL_AND] = inext;
// needle remainder: no = need boundary cell
// a: vvvvvvv
// b: yyyyyyyvvvvvvv
// r: yyyyyyy & 0 & vvvvvvv
if ( ar === 0 ) {
buf32[icell+CELL_AND] = this.addCell(inext, 0, 0);
iboundary = this.allocateCell();
buf32[icell+CELL_AND] = iboundary;
buf32[iboundary+CELL_AND] = inext;
return iboundary;
}
// needle remainder: yes = need new cell for remaining characters
// needle remainder: yes = need new cell for remaining
// characters
// a: wwwwvvvvvvv
// b: yyyyyyyvvvvvvv
// r: (0 & wwww | yyyyyyy) & vvvvvvv
else {
buf32[inext+CELL_OR] =
this.addCell(0, 0, this.addSegment(a, 0, ar));
buf32[icell+CELL_AND] = inext;
iboundary = this.allocateCell();
buf32[inext+CELL_OR] = this.addCell(
iboundary, 0, toSegmentInfo(aL0, 0, ar)
);
return iboundary;
}
}
return 1;
//debugger; // jshint ignore:line
}
}
optimize() {
this.shrinkBuf();
return {
byteLength: this.buf.byteLength,
byteLength: this.buf8.byteLength,
char0: this.buf32[CHAR0_SLOT],
};
}
@ -477,19 +564,117 @@ const SEGMENT_INFO = 2;
? decoder.decodeSize(selfie)
: selfie.length << 2;
if ( byteLength === 0 ) { return false; }
byteLength = byteLength + PAGE_SIZE-1 & ~(PAGE_SIZE-1);
if ( byteLength > this.buf.length ) {
this.buf = new Uint8Array(byteLength);
this.buf32 = new Uint32Array(this.buf.buffer);
byteLength = roundToPageSize(byteLength);
if ( byteLength > this.buf8.length ) {
this.buf8 = new Uint8Array(byteLength);
this.buf32 = new Uint32Array(this.buf8.buffer);
this.haystack = this.buf8.subarray(
HAYSTACK_START,
HAYSTACK_START + HAYSTACK_SIZE
);
}
if ( shouldDecode ) {
decoder.decode(selfie, this.buf.buffer);
decoder.decode(selfie, this.buf8.buffer);
} else {
this.buf32.set(selfie);
}
return true;
}
storeString(s) {
const n = s.length;
if ( (this.buf8.length - this.buf32[CHAR1_SLOT]) < n ) {
this.growBuf(0, n);
}
const offset = this.buf32[CHAR1_SLOT];
this.buf32[CHAR1_SLOT] = offset + n;
const buf8 = this.buf8;
for ( let i = 0; i < n; i++ ) {
buf8[offset+i] = s.charCodeAt(i);
}
return offset - this.buf32[CHAR0_SLOT];
}
extractString(i, n) {
if ( this.textDecoder === null ) {
this.textDecoder = new TextDecoder();
}
const offset = this.buf32[CHAR0_SLOT] + i;
return this.textDecoder.decode(
this.buf8.subarray(offset, offset + n)
);
}
// WASMable.
startsWith(haystackOffset, needleOffset, needleLen) {
if ( (haystackOffset + needleLen) > this.haystackSize ) {
return false;
}
const haystackCodes = this.haystack;
const needleCodes = this.buf8;
needleOffset += this.buf32[CHAR0_SLOT];
for ( let i = 0; i < needleLen; i++ ) {
if (
haystackCodes[haystackOffset+i] !==
needleCodes[needleOffset+i]
) {
return false;
}
}
return true;
}
// Find the left-most instance of substring in main string
// WASMable.
indexOf(haystackBeg, haystackEnd, needleOffset, needleLen) {
haystackEnd -= needleLen;
if ( haystackEnd < haystackBeg ) { return -1; }
const haystackCodes = this.haystack;
const needleCodes = this.buf8;
needleOffset += this.buf32[CHAR0_SLOT];
let i = haystackBeg;
let c0 = needleCodes[needleOffset];
for (;;) {
i = haystackCodes.indexOf(c0, i);
if ( i === -1 || i > haystackEnd ) { return -1; }
let j = 1;
while ( j < needleLen ) {
if ( haystackCodes[i+j] !== needleCodes[needleOffset+j] ) {
break;
}
j += 1;
}
if ( j === needleLen ) { return i; }
i += 1;
}
return -1;
}
// Find the right-most instance of substring in main string.
// WASMable.
lastIndexOf(haystackBeg, haystackEnd, needleOffset, needleLen) {
needleOffset += this.buf32[CHAR0_SLOT];
let i = haystackEnd - needleLen;
if ( i < haystackBeg ) { return -1; }
const haystackCodes = this.haystack;
const needleCodes = this.buf8;
let c0 = needleCodes[needleOffset];
for (;;) {
i = haystackCodes.lastIndexOf(c0, i);
if ( i === -1 || i < haystackBeg ) { return -1; }
let j = 1;
while ( j < needleLen ) {
if ( haystackCodes[i+j] !== needleCodes[needleOffset+j] ) {
break;
}
j += 1;
}
if ( j === needleLen ) { return i; }
i -= 1;
}
return -1;
}
//--------------------------------------------------------------------------
// Private methods
//--------------------------------------------------------------------------
@ -512,28 +697,15 @@ const SEGMENT_INFO = 2;
return icell;
}
addSegment(s, l, r) {
const n = r - l;
if ( n === 0 ) { return 0; }
const buf32 = this.buf32;
const des = buf32[CHAR1_SLOT];
buf32[CHAR1_SLOT] = des + n;
const buf8 = this.buf;
for ( let i = 0; i < n; i++ ) {
buf8[des+i] = s.charCodeAt(l+i);
}
return (n << 24) | (des - buf32[CHAR0_SLOT]);
}
growBuf(trieGrow, charGrow) {
const char0 = Math.max(
(this.buf32[TRIE1_SLOT] + trieGrow + PAGE_SIZE-1) & ~(PAGE_SIZE-1),
roundToPageSize(this.buf32[TRIE1_SLOT] + trieGrow),
this.buf32[CHAR0_SLOT]
);
const char1 = char0 + this.buf32[CHAR1_SLOT] - this.buf32[CHAR0_SLOT];
const bufLen = Math.max(
(char1 + charGrow + PAGE_SIZE-1) & ~(PAGE_SIZE-1),
this.buf.length
roundToPageSize(char1 + charGrow),
this.buf8.length
);
this.resizeBuf(bufLen, char0);
}
@ -546,46 +718,26 @@ const SEGMENT_INFO = 2;
}
resizeBuf(bufLen, char0) {
bufLen = bufLen + PAGE_SIZE-1 & ~(PAGE_SIZE-1);
if (
bufLen === this.buf.length &&
char0 === this.buf32[CHAR0_SLOT]
) {
bufLen = roundToPageSize(bufLen);
if ( bufLen === this.buf8.length && char0 === this.buf32[CHAR0_SLOT] ) {
return;
}
const charDataLen = this.buf32[CHAR1_SLOT] - this.buf32[CHAR0_SLOT];
if ( bufLen !== this.buf.length ) {
if ( bufLen !== this.buf8.length ) {
const newBuf = new Uint8Array(bufLen);
newBuf.set(
new Uint8Array(
this.buf.buffer,
0,
this.buf32[TRIE1_SLOT]
),
0
);
newBuf.set(
new Uint8Array(
this.buf.buffer,
this.buf32[CHAR0_SLOT],
charDataLen
),
char0
);
this.buf = newBuf;
this.buf32 = new Uint32Array(this.buf.buffer);
newBuf.set(this.buf8.subarray(0, this.buf32[TRIE1_SLOT]), 0);
newBuf.set(this.buf8.subarray(this.buf32[CHAR0_SLOT], this.buf32[CHAR1_SLOT]), char0);
this.buf8 = newBuf;
this.buf32 = new Uint32Array(this.buf8.buffer);
this.buf32[CHAR0_SLOT] = char0;
this.buf32[CHAR1_SLOT] = char0 + charDataLen;
this.haystack = this.buf8.subarray(
HAYSTACK_START,
HAYSTACK_START + HAYSTACK_SIZE
);
}
if ( char0 !== this.buf32[CHAR0_SLOT] ) {
this.buf.set(
new Uint8Array(
this.buf.buffer,
this.buf32[CHAR0_SLOT],
charDataLen
),
char0
);
this.buf8.copyWithin(char0, this.buf32[CHAR0_SLOT], this.buf32[CHAR1_SLOT]);
this.buf32[CHAR0_SLOT] = char0;
this.buf32[CHAR1_SLOT] = char0 + charDataLen;
}
@ -605,16 +757,24 @@ const SEGMENT_INFO = 2;
this.size = size;
}
add(s, i = 0) {
if ( this.container.add(this.iroot, s, i) === 1 ) {
add(i, n, pivot = 0) {
const iboundary = this.container.add(this.iroot, i, n, pivot);
if ( iboundary !== 0 ) {
this.size += 1;
return true;
}
return false;
return iboundary;
}
matches(a, i) {
return this.container.matches(this.iroot, a, i);
getExtra(iboundary) {
return this.container.buf32[iboundary+BCELL_EXTRA];
}
setExtra(iboundary, v) {
this.container.buf32[iboundary+BCELL_EXTRA] = v;
}
matches(i) {
return this.container.matches(this.iroot, i);
}
dump() {
@ -623,6 +783,10 @@ const SEGMENT_INFO = 2;
}
}
get $l() { return this.container.$l; }
get $r() { return this.container.$r; }
get $iu() { return this.container.$iu; }
[Symbol.iterator]() {
return {
value: undefined,
@ -646,7 +810,7 @@ const SEGMENT_INFO = 2;
let i0 = this.container.buf32[CHAR0_SLOT] + (v & 0x00FFFFFF);
const i1 = i0 + (v >>> 24);
while ( i0 < i1 ) {
this.charBuf[this.charPtr] = this.container.buf[i0];
this.charBuf[this.charPtr] = this.container.buf8[i0];
this.charPtr += 1;
i0 += 1;
}

3
src/js/traffic.js
View file

@ -221,7 +221,6 @@ const onBeforeRootFrameRequest = function(fctxt) {
url: requestURL,
hn: requestHostname,
dn: fctxt.getDomain() || requestHostname,
fc: logData.compiled,
fs: logData.raw
}));
@ -848,7 +847,7 @@ const injectCSP = function(fctxt, pageStore, responseHeaders) {
µb.staticNetFilteringEngine.matchAndFetchData(fctxt, 'csp');
for ( const directive of staticDirectives ) {
if ( directive.result !== 1 ) { continue; }
cspSubsets.push(directive.data);
cspSubsets.push(directive.getData('csp'));
}
// URL filtering `allow` rules override static filtering.

57
src/js/utils.js
View file

@ -48,7 +48,6 @@
for ( let i = 0, n = this._chars.length; i < n; i++ ) {
this._validTokenChars[this._chars.charCodeAt(i)] = i + 1;
}
// Four upper bits of token hash are reserved for built-in predefined
// token hashes, which should never end up being used when tokenizing
// any arbitrary string.
@ -62,10 +61,14 @@
this._urlIn = '';
this._urlOut = '';
this._tokenized = false;
this._tokens = [ 0 ];
this._tokens = new Uint32Array(1024);
this.knownTokens = new Uint8Array(65536);
this.resetKnownTokens();
this.MAX_TOKEN_LENGTH = 7;
this.charCodes = new Uint8Array(2048);
this.charCodeCount = 0;
}
setURL(url) {
@ -91,17 +94,24 @@
}
// Tokenize on demand.
getTokens() {
getTokens(encodeInto) {
if ( this._tokenized ) { return this._tokens; }
let i = this._tokenize();
i = this._appendTokenAt(i, this.anyTokenHash, 0);
let i = this._tokenize(encodeInto);
this._tokens[i+0] = this.anyTokenHash;
this._tokens[i+1] = 0;
i += 2;
if ( this._urlOut.startsWith('https://') ) {
i = this._appendTokenAt(i, this.anyHTTPSTokenHash, 0);
this._tokens[i+0] = this.anyHTTPSTokenHash;
this._tokens[i+1] = 0;
i += 2;
} else if ( this._urlOut.startsWith('http://') ) {
i = this._appendTokenAt(i, this.anyHTTPTokenHash, 0);
this._tokens[i+0] = this.anyHTTPTokenHash;
this._tokens[i+1] = 0;
i += 2;
}
i = this._appendTokenAt(i, this.noTokenHash, 0);
this._tokens[i] = 0;
this._tokens[i+0] = this.noTokenHash;
this._tokens[i+1] = 0;
this._tokens[i+2] = 0;
this._tokenized = true;
return this._tokens;
}
@ -136,13 +146,7 @@
// https://github.com/chrisaljoudi/uBlock/issues/1118
// We limit to a maximum number of tokens.
_appendTokenAt(i, th, ti) {
this._tokens[i+0] = th;
this._tokens[i+1] = ti;
return i + 2;
}
_tokenize() {
_tokenize(encodeInto) {
const tokens = this._tokens;
let url = this._urlOut;
let l = url.length;
@ -151,19 +155,27 @@
url = url.slice(0, 2048);
l = 2048;
}
encodeInto.haystackSize = l;
const knownTokens = this.knownTokens;
const vtc = this._validTokenChars;
let i = 0, j = 0, v, n, ti, th;
const charCodes = encodeInto.haystack;
let i = 0, j = 0, c, v, n, ti, th;
for (;;) {
for (;;) {
if ( i === l ) { return j; }
v = vtc[url.charCodeAt(i++)];
c = url.charCodeAt(i);
charCodes[i] = c;
v = vtc[c];
i += 1;
if ( v !== 0 ) { break; }
}
th = v; ti = i - 1; n = 1;
for (;;) {
if ( i === l ) { break; }
v = vtc[url.charCodeAt(i++)];
c = url.charCodeAt(i);
charCodes[i] = c;
v = vtc[c];
i += 1;
if ( v === 0 ) { break; }
if ( n === 7 ) { continue; }
th = th << 4 ^ v;
@ -292,7 +304,12 @@
this.properties = new Map();
}
push(args) {
this.block[this.block.length] = this.stringifier(args);
this.block.push(this.stringifier(args));
}
last() {
if ( Array.isArray(this.block) && this.block.length !== 0 ) {
return this.block[this.block.length - 1];
}
}
select(blockId) {
if ( blockId === this.blockId ) { return; }