Filter List Manager library core

Overview

This crate represents a library for managing AdGuard filter lists.

This library can:

• Fetch filter lists
• Store downloaded filter lists
• Perform filter list updates
• ... and more

Table of Contents

• How to build
• Filters analysis notes
  • List of meta tags that the library parses from filter content
  • List of filter preprocessor directives supported by the library
• Usage
  • Create and setup configuration for library facade
  • Data integrity protection
  • How to create and fill up filters database
  • Database scheme updates
  • Operations with custom filters
  • Get operations
  • Other (All) operations
• Cookbook
  • Miscellaneous filters/scripts collection

How to build

cargo build -p adguard-flm --locked from workspace root

For Windows builds you may need to build with libsqlite-bundled feature enabled:
cargo build -p adguard-flm --features rusqlite-bundled --locked

Filters analysis notes

List of meta tags that the library parses from filter content

• ! Title - Name of the filter.
• ! Description - Detailed description of the filter.
• ! Version - Current version of the filter.
• ! Expires - Filter expiration period. It will be converted into seconds. See the tests for an example If this field is missing in the metadata, the global value from the configuration will be used. Before updating the filter, the value will be checked and aligned to the lower boundary (3600) if it is less than this value.
• ! Homepage - Filter website/homepage.
• ! TimeUpdated - When this filter was updated in registry. Format: 2024-08-13T13:30:53+00:00.
• ! Last modified - Alias for TimeUpdated. Format: 2024-08-13T12:01:26.703Z. You can choose one format for both fields.
• ! Diff-Path - Differential updates information
• ! License - Link to filter license.
• ! Checksum - Filter's base64(md5-checksum). This checksum will be calculated and compared only for index filters. See the source here

List of filter preprocessor directives supported by the library

See AdGuard preprocessor directives

The library supports:

• !#include file_path - Includes contents of file into filter and processes it. file_path must be:
  • Absolute URL with the same origin as the parent filter.
  • Relative URL.
  • File URL (only if the parent filter's URL has file scheme).
• !#if/!#endif/!#else - Condition compilation directives. They can be nested. Supported tokens:
  • () - parentheses
  • true/false - boolean values
  • && || - AND/OR operators
  • ! - NOT operator
  • Literal compiler constant from configuration. For example, windows, mac, etc. It works like this: if the constant encountered is in the configuration.compiler_conditional_constants list, then the condition becomes true, false otherwise

See the tests for more information:

• All directives
• !#include
• !#if / !#endif / !#else

Usage

Create and setup configuration for library facade

// Every instance of FilterListManager must have its own configuration
let mut configuration = Configuration::default();

// Sets urls for filters indices.
configuration.metadata_url = "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/refs/heads/master/platforms/extension/safari/filters.json".to_string();
configuration.metadata_locales_url = "https://raw.githubusercontent.com/AdguardTeam/FiltersRegistry/refs/heads/master/platforms/extension/safari/filters_i18n.js".to_string();

// Sets locale. Will be used for returning localized strings for filters,
// groups, and tags, where applicable.
configuration.locale = "pt_PT".to_string();

// Sets app name and version for user-agent header.
// Required fields.
configuration.app_name = "FlmApp".to_string();
configuration.version = "1.2.3".to_string();

// Creates facade instance
let flm = FilterListManagerImpl::new(configuration)?;

Example references

Configuration reference
FilterListManager reference

Data integrity protection

The library supports cryptographic integrity protection for stored filter data using BLAKE3 keyed hashing. When enabled, the library signs and verifies:

• Filter rules — the body of each filter's rule list and its !#include parts.
• Filter metadata — 10 critical fields per filter (filter_id, download_url, subscription_url, is_trusted, is_enabled, is_installed, version, last_update_time, last_download_time, expires).
• Filter count — a signed total number of filter rows stored in the metadata table, protecting against unauthorized addition or removal of filters.

A lightweight count-signature check (one metadata read + one COUNT(*) + one BLAKE3 hash) runs automatically at the start of every facade method that reads or writes filter data. Per-entity metadata and rules signatures are verified on the entities actually returned to the caller. A full streaming verification of all signatures can be triggered explicitly via verify_integrity().

The typical workflow is to sign the database once during initial setup or when enabling integrity protection for existing data. After that, the library automatically signs all newly installed or updated filters, maintaining integrity protection transparently without requiring manual intervention.

To enable integrity protection:

use adguard_flm::{Configuration, FilterListManagerImpl, generate_random_key};

// 1. Generate a cryptographically secure random key
let integrity_key = generate_random_key()?;

// 2. Create configuration with the integrity key
let mut config = Configuration::default();
config.app_name = "MyApp".to_string();
config.version = "1.0.0".to_string();
// This enables integrity protection
config.integrity_key = Some(integrity_key);

To sign storage for the first time:

// This call has rules:
// - Call this only if your data is not signed yet, or you have instantiated
//   the FLM with a new integrity key
// - If integrity protection is disabled, this call will return an error
// - If integrity protection is enabled, you should call this method
//   immediately after creating the FLM instance and before any other
//   operations
flm.sign_all_data()?;

To verify integrity:

// Performs a full streaming verification of all signatures (rules, includes,
// metadata, count). Returns FilterIntegrityCheckFailed(filter_id) if any
// signature is invalid, or FilterIntegrityCheckFailed(0) for a count mismatch.
flm.verify_integrity()?;

To rotate the integrity key:

// Generate a new key and re-sign all data atomically
let new_key = generate_random_key()?;
flm.sign_all_data_with_new_key(new_key)?;

[!IMPORTANT] Once integrity protection is enabled, you must call sign_all_data() immediately after creating the FLM instance and before any other operations, or every subsequent call will fail with FilterIntegrityCheckFailed(filter_id). All subsequent filter installations and updates are automatically signed.

How to create and fill up filters database

// Creates and configures the database. Populates the database with information
// from the filter indexes (filters metadata), the paths to which are specified
// in the configuration.
// In addition, this method applies migrations that have not yet been applied.
// See the lift_up_database method for details on "lifting" a database.
// Note, should be used once a week or less.
flm.pull_metadata()?;

// Then, downloads the contents of the filters.
// !Note! should be used no more than once an hour.
flm.update_filters(true, 0, true)?;

[!NOTE] By default, the application operates with a database located in the current working directory (cwd).
The database file name is generated based on the format agflm_{configuration.filter_list_type.to_string().to_lowercase()}.db.
For standard filters, the file path will be $CWD/agflm_standard.db.

Database scheme updates

Database schema updates (migrations) are possible using the flm.lift_up_database() method. The method “raises” the state of the database to the working state.

If the database doesn't exist:

• Creates database
• Rolls up the schema
• Rolls migrations
• Performs bootstrap.

If the database is an empty file:

• Rolls the schema
• Rolls migrations
• Performs bootstrap.

... and so on.

Migrations notes

Starting with version 0.7.1 the database is “uplifted” automatically when the filter_list_manager constructor is called. To override this behavior you need to disable it in the configuration: configuration.auto_lift_up_database = false;.\

Note: methods flm.update_filters(), flm.force_update_filters_by_ids()
should be used no more than once an hour, method flm.pull_metadata()
should be used no more than once a week.

Storage notes

[!IMPORTANT] Database lifting
If you have disabled automatic lifting, you must invoke it yourself after each library update if you don't want to miss a migration.

[!CAUTION] SQLITE_BUSY Error
The library ensures that when using a single FLM instance for a single database file (also, by default, a database type) in a multithreaded environment, database queries will not return SQLITE_BUSY errors.

Operations with custom filters

The library categorizes all filters into three types:

1. Index Filters - Filters created by parsing the index (registry).
2. Custom Filters - Filters added (and edited) by the user using the library's methods.
3. Special Filters - Custom filters preconfigured by the library's scripts.

You can refer to the db constants file to check the indicators for special and custom filters.

// Installs a custom filter.
let custom_filter = flm.install_custom_filter_list(
    String::from("https://example.com/custom_filter.txt"),
    true, // The filter list is marked as trusted.
    Some(String::from("Custom title")),
    Some(String::from("Custom description"))
).unwrap();

// Edit metadata.
flm.update_custom_filter_metadata(
    custom_filter.id,
    String::from("new title"),
    false // The filter list is marked as not trusted.
).unwrap();

// Turn on this filter.
flm.enable_filter_lists(vec![custom_filter.id], true);

// Remove this filter.
flm.delete_custom_filter_lists(vec![custom_filter.id]);

Installing a custom filter from a string instead of downloading it

let string_contents = String::from(r"
! Checksum: ecbiyIyplBZKLeNzi64pGA
...
! JS API START
#%#var AG_onLoad=function(func){if(document.readyState==="complete"||document.readyState==="interactive")func();else
... 
");
flm.install_custom_filter_from_string(
    String::new(), // download url
    1719505304i64, // last_download_time value. Explanation: Can we update filter? Answer: (filter.last_download_time + filter.expires < now()) 
    true, // Enabled
    true, // Trusted
    string_contents, // Filter body
    None, // Filter title - Option<String>
    None  // Filter description - Option<String>
);

Save operations for custom filters rules

// Saves the structure containing the filter rules.
flm.save_custom_filter_rules(/* FilterListRules */ rules_for_new_local_custom_filter);

// You can save only the disabled rules for a filter list 
flm.save_disabled_rules(filter.id, /* Vec<String> */ disabled_rules_list);

Example references

FilterListRules reference

Get operations

// Retrieves filter metadata by its ID from the database **with** its rules.
// Returns Optional<FullFilterList>.
flm.get_full_filter_list_by_id(id /* FilterId */);

// Retrieves all enabled filters as ActiveRulesInfo.
flm.get_active_rules();

// Gets a list of [`ActiveRulesInfoRaw`] from filters with `filter.is_enabled=true` flag.
// `filter_by` - If empty, returns all active rules, otherwise returns the intersection between `filter_by` and all active rules
flm.get_active_rules_raw(filter_by /* Vec<FilterId> */);

// Retrieves all filters metadata from the database **without** their rules.
// Returns Vec<StoredFilterMetadata>
flm.get_stored_filters_metadata();

// Retrieves filter metadata by its ID from the database **without** its rules.
// Returns Optional<StoredFilterMetadata>.
flm.get_stored_filter_metadata_by_id(id /* FilterId */);

// Retrieves a list of FilterListRulesRaw by IDs.
// This method acts in the same way as the `IN` database operator. Returns only the entities that are found
flm.get_filter_rules_as_strings(ids /* Vec<FilterId> */);

// Reads the rule list for a specific filter in chunks, applying exceptions from the disabled_rules list on the fly.
// The default size of the read buffer is 1 megabyte. But this size can be exceeded if a longer string appears in the list of filter rules.
// The main purpose of this method is to reduce RAM consumption when reading large size filters.
flm.save_rules_to_file_blob(id /* FilterId */, file_path /* String or AsRef<Path> */);

// Returns lists of disabled rules by list of filter IDs as Vec<DisabledRulesRaw>
flm.get_disabled_rules(ids /* Vec<FilterId> */);

// Fetches filter list by URL and returns its raw metadata.
// Returns FilterListMetadata.
flm.fetch_filter_list_metadata(url /* String */);

// Fetches filter list by URL and returns its raw metadata and body.
// Returns FilterListMetadataWithBody.
flm.fetch_filter_list_metadata_with_body(url /* String */);

// Returns lists of rule counts by list of filter IDs as Vec<RulesCountByFilter>
flm.get_rules_count(ids /* Vec<FilterId> */);

These example references

FullFilterList reference
StoredFilterMetadata reference
ActiveRulesInfo reference
ActiveRulesInfoRaw reference
FilterListRulesRaw reference
DisabledRulesRaw reference
RulesCountByFilter reference

Other (All) operations

Facade Interface

Cookbook

Miscellaneous filters/scripts collection

By setting configuration.filter_list_type = FilterListType::MISC, you can create a dedicated FLM instance with its own database (for example, agflm_misc.db) and a custom index.

This allows you to:

• keep a collection of unrelated filters/scripts in a single place
• store arbitrary custom metadata in the index
• use stable, hard-coded IDs for your filters/scripts
• still use all FLM features (storage, updates, etc.)

Here is an index example. You may also be interested in the consistency checker (index consistency rules) and the internal index entities (FilterIndexEntity).

use adguard_flm::{Configuration, FilterListManager, FilterListManagerImpl, FilterListType};

const FILTER_A: FilterId = 1;
const FILTER_B: FilterId = 3;

let mut configuration = Configuration::default();

// Sets the URL of your custom index (can be a local `file://` URL).
configuration.metadata_url = "file:///path/to/filters.json".to_string();
// Optional: localizations index. Leave empty if you don't have one.
configuration.metadata_locales_url = String::new();

// Use a dedicated database for miscellaneous filters/scripts.
configuration.filter_list_type = FilterListType::MISC;

// Required fields.
configuration.app_name = "FlmApp".to_string();
configuration.version = "1.2.3".to_string();

let flm = FilterListManagerImpl::new(configuration)?;
// Sync metadata from the index into the database.
flm.pull_metadata()?;
// Download/update filter contents.
flm.update_filters(false, 0, false)?;
// Gets your known filters contents
flm.get_filter_rules_as_strings(vec![FILTER_A, FILTER_B])?;
// Gets "Filter_A" metadata
flm.get_stored_filter_metadata_by_id(FILTER_A)?;