This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
advanced-adblock [2022/01/20 15:43] – [Adblock]-formatting hogwild | advanced-adblock [2023/08/06 16:14] – [Adblock Settings] -accuracy, condense hogwild | ||
---|---|---|---|
Line 1: | Line 1: | ||
====== Adblock ====== | ====== Adblock ====== | ||
- | The Adblock | + | This menu contains settings to configure FreshTomato' |
- | FreshTomato' | + | In this documentation, |
- | If only elements are blocked, it's likely the page will load without error -- uncluttered. | + | |
- | )) | + | |
- | Alternatively, | + | ===== v1 and v2 ===== |
- | For Adblock | + | There are currently two versions of Adblock: [[advanced-adblock# |
- | There are three optional settings that help ensure smooth | + | \\ This table lists the Adblock |
- | | + | ^ Hardware ^ FT < 2023.1 ^ FT >= 2023.1 ^ |
- | - Enabling Advanced/[[dhcp_dns|DHCP/DNS]]/Prevent Client auto DoH | + | | MIPS R1 | [[advanced-adblock# |
- | - Enabling the "DoH Server" | + | | MIPS R2 | [[advanced-adblock# |
+ | | ARM | [[advanced-adblock# | ||
- | Warning: FreshTomato' | + | \\ Adblock v1 functionality |
+ | Adblock v2 uses advanced methods to block ads. It should be the preferred choice whenever possible. | ||
+ | |||
+ | ===== How Adblock Works ===== | ||
+ | |||
+ | FreshTomato' | ||
+ | |||
+ | * dnsmasq resolves the domain' | ||
+ | * The adblock script then replaces that address with an address of 0.0.0.0 . | ||
+ | * The 0.0.0.0 address is sent to the client requesting DNS resolution. | ||
+ | * Since 0.0.0.0. is an NXDOMAIN (invalid/ | ||
+ | |||
+ | Given a list of sources, the original script simply blocks ads. However, there are other reasons for your network to avoid communicating with certain servers. This function was therefore renamed "DNS Filtering" | ||
===== Adblock Settings ===== | ===== Adblock Settings ===== | ||
- | **Enable**: Checking this box enables FreshTomato' | + | |
- | **Debug Mode: **Checking this box enables debug mode in the log. This tells FreshTomato that you want all DNS queries that are routed to dnsmasq | + | The adblock script downloads lists of URLs/ |
- | [[https:// | + | \\ |
- | ===== Blacklist URL ===== | + | **Debug mode (v1)** |
- | This section | + | Checking this box enables debug mode for dnsmasq in the log. This tells FreshTomato that you want all DNS queries routed to dnsmasq to be logged to the system log. This is useful when testing/ |
- | **On:** Clicking on one of the blacklist rows will make a checkbox appear at the far left of the row. Checking that box will enable the download (and update) and usage of that particular DNS blacklist. When you are finished selecting which blacklists you wish to use, click Save for the changes to take effect. | + | \\ |
- | **Blacklist URL: **Shows the location on the Internet where that particular blacklist can be found. | + | {{:: |
- | **Description:** Display a name (if the creator used one) for the particular blacklist. | + | \\ |
+ | |||
+ | **Max Log Level (v2)**: | ||
+ | |||
+ | Thew newer v2 interface lets you set the maximum log level output the script will generate. | ||
+ | |||
+ | Supported levels include: | ||
+ | |||
+ | | ||
+ | | ||
+ | * 4 | ||
+ | * 5 | ||
+ | * 6 | ||
+ | * 7 (Debug level) writes very detailed log information. This is helpful when troubleshooting common problems. | ||
+ | |||
+ | The higher the setting, the more detailed will be the information recorded in the logs. | ||
\\ | \\ | ||
- | [[https:// | + | {{:: |
\\ | \\ | ||
- | **Delete: **Clicking this button on a checked Blacklist URL will permanently delete that Blacklist. Note that there is no option | + | A good way to view the logs is to go to the [[status-log|Logs]] menu and enter ' |
- | **Add:** Clicking Add inserts a blank row in which you can type a new URL from which to download and use a new Blacklist. | + | {{:: |
- | An Autoupdate function will launch randomly every day between 2:00AM and 2:59 to download the most up-to-date Blacklists from the URLs in the list. | + | **Blockfile size limit (v2 only)**: |
+ | Adblock v1 may crash FreshTomato if it loads lists whose combined information exceeds your router' | ||
- | ===== Custom Blacklist ===== | + | \\ |
- | The Custom Blacklist section contains a field into which you enter custom blocking entries. All entries must be separated by spaces for the function to work properly for each entry. | + | {{:: |
- | ===== Custom Whitelist ===== | + | \\ |
- | Here you enter custom URLs that you would like to allow, by default. Entry rule are similar | + | This limit is calculated as 10% of physical RAM (when external storage is set). When no external storage is found, the limit is calculated as 6.5% of RAM. The limit can be also manually configured. However, if your device becomes unstable, it is advised |
- | ===== Adblock Notes ===== | + | To be clear, this is a limit, not a target. There' |
- | Testing/ | + | \\ |
- | Blacklisted | + | **Custom Path (v2):** |
+ | |||
+ | An important v2 feature is the option to configure a path to permanent storage where adblock can store relevant files. It is strongly advised to specify a custom path. A custom path enables extra functionality in the script. It allows adblock to store whole lists, their http headers and the actual compiled blockfile on external storage. This offloads RAM demand (/tmp) to permanent storage, and allows for information to survive a reboot. | ||
+ | |||
+ | \\ {{adblock-v2-custom_path.jpg? | ||
+ | |||
+ | \\ | ||
+ | |||
+ | This is very useful when the script is re-run (manually | ||
+ | |||
+ | When a custom path is defined, adblock will: | ||
+ | |||
+ | - Download all http headers from only the enabled lists | ||
+ | - Compare those headers with the ones stored locally | ||
+ | - Decide whether to re-process the blocklists or not. | ||
+ | |||
+ | This runs as follows: | ||
+ | |||
+ | * If the configuration wasn't changed and; | ||
+ | * No updated lists are available | ||
+ | * Skip the re-processing and return to idle. | ||
+ | |||
+ | This saves time and resources. | ||
+ | |||
+ | Here, configuration means any externally-mapped files where // | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ==== Domain Blacklist URLs & Group-of-Lists ==== | ||
+ | |||
+ | This table contains a list of the blacklists FreshTomato can download and use to block ads. | ||
+ | |||
+ | **On: | ||
+ | |||
+ | **Blacklist URL: | ||
+ | |||
+ | **NOTE:** Since release 2023.4 by default, no blocklists were defined as part of the standard releases. This was done to to reduce NVRAM demand. However, you can add your preferred lists manually. A good summary of these lists is the official [[adblock_dns_filtering|Adblock (DNS filtering) lists]] page on this Wiki. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{:: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | **Description: | ||
+ | |||
+ | **Delete: | ||
+ | |||
+ | There is no option to reset these to the original Blacklist URL entries. | ||
+ | |||
+ | If you delete a URL that's important to you, you will need to: | ||
+ | |||
+ | * Re-enter it into the Blacklist URL table or; | ||
+ | * Reset FreshTomato to default settings | ||
+ | |||
+ | **Add: | ||
+ | |||
+ | You can also add a comment in the Description field. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ==== Compatible data formats ==== | ||
+ | |||
+ | Target lists to be downloaded must be in plain text format. As above, v2 can extract domains from lists in a variety of formats. Essentially, | ||
+ | |||
+ | **Group-of-lists** | ||
+ | |||
+ | Currently, v2 also accepts a new list format called " | ||
+ | |||
+ | \\ | ||
+ | |||
+ | This can be visualized as follows: | ||
+ | |||
+ | <code -> | ||
+ | URL |--> list | ||
+ | </ | ||
+ | |||
+ | | ||
+ | |||
+ | <code -> | ||
+ | URL |--> Group-of-lists-file |--> URL |--> list | ||
+ | | ||
+ | | ||
+ | | ||
+ | </ | ||
+ | |||
+ | | ||
+ | |||
+ | Adblock automatically handles EOL (End of Line) characters in files, when necessary. It converts them internally. | ||
+ | |||
+ | Group-of-lists are visible in the log file. For example, they might appear as: " | ||
+ | |||
+ | |||
+ | ==== Domain blacklist custom ==== | ||
+ | |||
+ | {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | === Domain Syntax === | ||
+ | |||
+ | Several syntaxes are valid for this list: | ||
+ | |||
+ | * Standard domains (one entry per line) | ||
+ | * A path to a local file where domains are defined. The file should contain one domain per line. \\ For example: "/ | ||
+ | * A domain prefixed with a " | ||
+ | |||
+ | \\ | ||
+ | |||
+ | When adding custom entries: | ||
+ | |||
+ | If you blacklist: " | ||
+ | |||
+ | " | ||
+ | |||
+ | " | ||
+ | |||
+ | and all other subdomains will be blacklisted. | ||
+ | |||
+ | However, this may not be what you want. | ||
+ | |||
+ | To prevent subdomains of an entry from being filtered, prepend a " | ||
+ | |||
+ | For example, if you enter " | ||
+ | |||
+ | Prepending a " | ||
+ | |||
+ | Duplicate or redundant entries make for a larger blockfile size, which can increase processor cycles, RAM usage and storage space. Thus, pruning significant amounts of redundant entries can reduce CPU, memory and storage needs. | ||
+ | |||
+ | Any line starting with a "#" | ||
+ | |||
+ | **Sort a-z ↓** **:** Clicking this button sorts the contents of this field alphabetically. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ==== Domain whitelist ==== | ||
+ | |||
+ | {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Domain whitelist can use: | ||
+ | |||
+ | * Standard domains (one entry per line). | ||
+ | * A path to a file of domains (one domain per line) . \\ For example: "/ | ||
+ | * Prepending a " | ||
+ | * Whitelisting all subdomains may not be your goal. Whitelisting " | ||
+ | |||
+ | When adding custom entries: | ||
+ | |||
+ | If you whitelist: " | ||
+ | |||
+ | " | ||
+ | |||
+ | " | ||
+ | |||
+ | and all other subdomains will be whitelisted. | ||
+ | |||
+ | However, this may not be what you want. | ||
+ | |||
+ | To prevent subdomains of an entry from being whitelisted, | ||
+ | |||
+ | For example, if you enter " | ||
+ | |||
+ | Prepending a " | ||
+ | |||
+ | Any line starting with a "#" | ||
+ | |||
+ | **Sort a-z ↓** : Clicking this button sorts the field content alphabetically. | ||
+ | |||
+ | | ||
+ | |||
+ | === Maintaining the Domain whitelist === | ||
+ | |||
+ | A good way to maintain the whitelist is to share it on your LAN, and educate users about nslookup verification and whitelist additions. | ||
+ | |||
+ | Such a process could be automated as follows: | ||
+ | |||
+ | - Map a file in the Whitelist section of the Adblock menu | ||
+ | - Share the file/folder via Samba, in the [[nas-samba|File Sharing]] menu | ||
+ | - Teach users: | ||
+ | - How to use nslookup on a URL to verify DNS lookups are being poisoned | ||
+ | - Where to find the whitelist (via its samba share) and; | ||
+ | - After the whitelist has been edited, to configure settings in [[admin-buttons|Buttons/ | ||
+ | |||
+ | | ||
+ | |||
+ | |||
+ | ==== Enforcing Client Compliance ==== | ||
+ | |||
+ | For Adblock to work properly, client devices **must be configured** to use FreshTomato' | ||
+ | |||
+ | For the latter, first enable DHCP in the [[basic-network|Network]] menu. To enable the DNS server, select "Use internal DNS" under in the [[advanced-dhcpdns|DHCP/ | ||
+ | |||
+ | These steps are mandatory. Clients that bypass FreshTomato' | ||
+ | |||
+ | Three optional settings help ensure proper Adblock operation: | ||
+ | |||
+ | - Enable //Intercept DNS port// in the [[advanced-dhcpdns|DHCP/ | ||
+ | - Enable //Prevent Client auto DoH// in the [[advanced-dhcpdns|DHCP/ | ||
+ | - Enable the "DoH Server" | ||
+ | |||
+ | Adblock v1 functionality is a reduced version of ad-blocking scripts taken from code outside FreshTomato. Enabling one of those outside scripts and FreshTomato Adblock at the same time may cause conflicts. Do not enable both. Adblock v2 uses advanced methods to block ads. It should be the preferred choice whenever possible. | ||
+ | |||
+ | Adblock/DNS Filtering affects name resolution only. If an application communicates directly via IP address, Adblock cannot prevent that. | ||
+ | |||
+ | Thorough management of domain blocking can be tedious work. For example, with email spam, you'll probably have to deal with false positives. | ||
+ | |||
+ | Regardless of version, the script is always named " | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ===== Adblock v2 ===== | ||
+ | |||
+ | Adblock v2 improves on the original (v1) script. V2 adds a richer feature set, with increased user interaction via a toolbar and command line options. | ||
+ | |||
+ | \\ \\ \\ {{: | ||
+ | |||
+ | |||
+ | ==== v2 Improvements ==== | ||
+ | |||
+ | * Adblock v2 performs similar basic DNS filtering functionality to v1, but in a more precise and controlled way. | ||
+ | * Allows you to run certain script functions without having to run all of them, to save time. | ||
+ | * Can accept different list formats (in plain text), and extract domains from files with different layouts. \\ This includes Easylist format. There may be false positives using Easylist format. For details, see: [[https:// | ||
+ | * Prevents resource starvation. Also, before running, v2 assesses its configuration for blocklist capacity. It's basically self-diagnosing and healing. | ||
+ | * Supports Quick-run. This mode allows you to add a domain to the lists without needing to reprocess (download) entire lists again. | ||
+ | * Supports external storage. Now, lists and headers are stored in optional permanent storage to help decide which lists to download. | ||
+ | * Prevents false positives via a new, hard-coded 30-minute interval between runs. This is enabled after an update is completed. | ||
+ | * Includes more troubleshooting options. Debug logging and script tracing can be enabled to look deep into the \\ lowest operational level of the script. | ||
+ | * Can be operated from the web interface or at a command line. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | To get help with adblock at the command line, type '' | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ {{: | ||
+ | |||
+ | | ||
+ | |||
+ | \\ \\ To open the last trace file automatically, | ||
+ | |||
+ | \\ {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ===== Using the v2 Controls ===== | ||
+ | |||
+ | \\ | ||
+ | |||
+ | The Controls button bar in the Advanced section let you control adblock script execution. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | | {{ : | ||
+ | | ::: | **Unload: | ||
+ | | ::: | **Update: | ||
+ | | ::: | **Reset-limit: | ||
+ | | ::: | **Clear all files: | ||
+ | | ::: | **Snapshot: | ||
+ | | ::: | **Enable Only: | ||
+ | | ::: | **Disable Only:** This lets you disable adblock ('' | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Refresh: There is now a refresh command at the bottom of the table. Refreshes are limited to a minimum frequency of 5 seconds. The more frequently you run refreshes, the higher the system load will climb. | ||
+ | |||
+ | |||
+ | ===== Quick-run (v2) and Full-run Operations ===== | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Modifications to the adblock script configuration can be categorized as follows: | ||
+ | |||
+ | - List (enable/ | ||
+ | - List content update | ||
+ | - Change of parameters (loglevel / blockfile limit / path ) | ||
+ | - **Addition of a simple blacklist_custom** | ||
+ | - **Addition of a simple whitelist** | ||
+ | - Addition of pruning blacklist_custom | ||
+ | - Addition of a strict whitelist | ||
+ | - Removal of a blacklist_custom | ||
+ | - Removal of a whitelist | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Only two operations from the above list can be run in Quick-run mode, as they don't require full processing: | ||
+ | |||
+ | * No. (4) Addition of simple custom blacklist domain/s | ||
+ | * No. (5) Addition of simple whitelisted domain/s | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Operations (4) and (5) are the most common actions performed when maintaining adblock script configuration. Quick-run operations are performed live on a running blockfile. Therefore, the adblock script must be loaded in order to run functions (4) or (5). A quick-run can be completed in literally seconds. A full-run may take minutes to complete. Any operations other than (4) or (5) above require a full-run. | ||
+ | |||
+ | Please note that " | ||
+ | |||
+ | |||
+ | ===== Command line operations (v2) ===== | ||
+ | |||
+ | | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | As seen above in the " | ||
+ | |||
+ | \\ Help | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Status | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Start | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Stop | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Update | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Upgrade | ||
+ | |||
+ | '' | ||
+ | |||
+ | Since it is stored in RAM, it will be lost after a reboot. Therefore, an extra parameter is supported: | ||
+ | |||
+ | \\ Test | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Reset | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Clear/ | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Trace/ | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Snapshot | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Enable | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ Disable | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ==== Adblock test command (v2) ==== | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{: | ||
+ | |||
+ | To simplify and speed up testing of broken resolution or other DNS issues, use the " | ||
+ | |||
+ | Specifically: | ||
+ | |||
+ | * dnsmasq answer - Displays the actual name resolution as seen by the router and LAN users. \\ The word " | ||
+ | * Cloudflare answer - This query bypasses the usual dnsmasq process and sends a query to address 1.1.1.1 . \\ This can help you to verify if a domain exists. | ||
+ | * Blockfile ref - This checks whether there are any references to the tested | ||
+ | |||
+ | \\ With these three pieces of information, | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | |||
+ | ===== Automatic list updates ===== | ||
+ | |||
+ | As part of normal adblock operation, a crontable entry is added to perform a daily list update. Update times are now randomized to prevent DDoS of the list providers' | ||
+ | |||
+ | \\ | ||
+ | |||
+ | To verify the current scheduled update time: | ||
+ | |||
+ | '' | ||
+ | |||
+ | \\ | ||
+ | |||
+ | In this example, it's **28** (min) **4** (am) * (Every day) * (Every month) * (Every day of the week). | ||
+ | |||
+ | This picture illustrates the syntax of crontable expressions: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{:: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | For help automating/ | ||
+ | |||
+ | If you stop adblock, the crontable expression is removed. This is by design. | ||
+ | |||
+ | |||
+ | ===== Future development goals ===== | ||
+ | |||
+ | * Add an option to let FreshTomato users share their Whitelists. \\ This would allow the creation of a shared database of " | ||
+ | * Add a test function to the web interface (currently only command line) | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ===== Adblock Notes and Troubleshooting ===== | ||
+ | |||
+ | ==== Troubleshooting v1 ==== | ||
+ | |||
+ | You can test Blacklisted or Custom entries | ||
+ | |||
+ | For example, this entry shows that dnsmasq replaced the true IP address of hbx.media.net with 0.0.0.0. | ||
'' | '' | ||
- | shows that the dnsmasq daemon replaced the true IP address of hbx.media.net with 0.0.0.0. Adblocker | + | Adblock |
- | If the router crashes, you may have used too many large Blacklists, | + | If your router crashes, you may be using too many large Blacklists, |
+ | |||
+ | There are websites available that allow you to test the effectiveness of your Adblock configuration. When doing so, please ensure you don't have any adblock/ | ||
+ | |||
+ | * [[https:// | ||
+ | * [[https:// | ||
\\ | \\ | ||
Line 75: | Line 551: | ||
**Adblock doesn' | **Adblock doesn' | ||
- | Increasingly, | + | Increasingly, |
+ | |||
+ | As mentioned earlier, enabling the "DoH Server" | ||
+ | |||
+ | \\ | ||
+ | |||
+ | |||
+ | ==== Troubleshooting v2 ==== | ||
+ | |||
+ | For troubleshooting, | ||
+ | |||
+ | In order to learn what might be going wrong, it's crucial to understand adblock' | ||
+ | |||
+ | * System load is too high in the 5min+ section (For example, > 1.5) | ||
+ | * RAM usage is too high (> 90%) | ||
+ | * dnsmasq ownership remains root (non-operational) until lists are fully loaded. It's normal to see " | ||
+ | * Too many dnsmasq restarts | ||
+ | * dnsmasq restart time is too high (> 15 seconds) | ||
+ | * adblock calls today | ||
+ | * last run errors | ||
+ | |||
+ | \\ | ||
+ | |||
+ | If the Blockfile size exceeds the //Blockfile size limit// value, the blockfile size will be reduced. If (with limit manually set high) dnsmasq still struggles to run properly after this, the safeDnsmasqRestart() process will reduce the Blockfile size in increments of 5% and reattempt to fully start dnsmasq after each reduction. When dnsmasq functions properly, the new auto-calculated limit is set as the current //Blockfile size limit//. | ||
+ | |||
+ | \\ | ||
+ | |||
+ | When a trace is running, | ||
+ | |||
+ | | ||
+ | |||
+ | \\ \\ To open the last trace file automatically, | ||
+ | |||
+ | \\ {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | \\ | ||
+ | |||
+ | **Apple iOS devices (paid iCloud service only)** | ||
+ | |||
+ | Apple IOS devices have settings that may interfere with adblock operations. The "Limit Address Tracking" | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{: | ||
+ | |||
+ | \\ {{: | ||
+ | |||
+ | \\ | ||
+ | |||
+ | Alternatively, | ||
+ | |||
+ | \\ | ||
+ | |||
+ | {{: | ||
+ | |||
+ | You might connect your mobile/laptop to different wireless networks at different locations. For this reason, it makes sense to disable Private Relay on a per WLAN basis. It is advised that you disable it wherever FreshTomato provides your Internet connection and Adblock is running. | ||
- | As mentioned earlier, enabling the "DoH Server" | + | Remember, these options are only available for paid iCloud customers. |