Site Tools


advanced-adblock

Adblock (DNS filtering)

This menu contains settings to configure FreshTomato's ad blocker-DNS filtering. This feature can block ads, some malware, crypto miners, privacy-invading scripts and more, depending on the lists you use.

You are strongly advised to read the How Adblock Works section to understand its functioning.

In documentation, the function/menu will be spelled capitalized (“Adblock”). The actual script will be spelled “adblock”, in lower case.

v1 and v2

There are currently two versions of Adblock: adblock_v1 and adblock_v2.


This table lists the Adblock versions available on different types of hardware

Hardware FT < 2023.1 FT >= 2023.1
MIPS R1 v1 v1
MIPS R2 v1 v2
ARM v1 v2


Adblock v1 functionality is taken from ad-blocking scripts outside FreshTomato. Do not enable one of those scripts and FreshTomato Adblock at the same time. Doing so may cause conflicts.

Adblock v2 uses advanced methods to block ads. It should be the preferred choice whenever possible.

How Adblock Works

Adblock works through DNS cache poisoning, as follows:


  1. A client makes a DNS request to resolve a (sub)domain
  2. Dnsmasq searches both its caches for a match of the requested (sub)domain.
    One cache is for positive lookups, and one is for failed lookups. 1)
    If an entry is found in either cache, the script proceeds to step 5.
  3. Dnsmasq searches its configuration files for a corresponding entry.
    For adblock, the “dnsmasq.adblock” file holds the (sub)domains to block.
    If found, the entry is processed. In the case of adblock, the action is:
    resolve the domain to “0.0.0.0” or “NXDOMAIN” (a non-existent domain).
    Dnsmasq then proceeds to step 5.
  4. Dnsmasq queries an upstream DNS server for the (sub)domain, directly
    or via stubby/dnscrypt. The result is a positive lookup, or an “NXDOMAIN”.
  5. The result from these steps is sent to the cache and returned
    to the requesting client.


Given a list of sources, the original script simply blocks ads. However, you may have other reasons for your network to to avoid communicating with certain servers. As a result, the function was renamed “Adblock-DNS Filtering” to reflect its increased abilities.

Adblock Settings


Enable: Checking this box turns on Adblock.

The adblock script downloads lists of URLs/domains to block, mostly from the Blacklist URL & Group-of-lists table. You can select or deselect any list in this table to be included in the downloads (the “blocklist”).


Debug mode (v1 only)

Checking this enables debug mode for dnsmasq in the log. This tells FreshTomato to log all DNS queries routed to dnsmasq to the system log. This is useful when testing/troubleshooting Adblock. Because this option is identical to “Debug Mode” in the DHCP/DNS/TFTP menu, it was removed from the Adblock menu in v2.

For more on testing, see the Notes and Troubleshooting section of this page.



Max Log Level (v2):

The newer v2 interface lets you set the maximum log level output the script will generate.

Options include:

  • 0: The script will log only basic messages like start and stop.
  • 3 (Error). Once you're confident with your setup, this setting is sufficient. (Default).
  • 4
  • 5
  • 6
  • 7 (Debug level) writes very detailed log information. This is helpful for troubleshooting.


The higher the number, the more detailed the information the logs will record.




A good way to view Adblock log entries is to go to the Logs menu and type “adblock” in the “Find in syslog” field. This filters the view to show only adblock entries.

When set to level 7, adblock writes additional debug files. These are saved in “/tmp/adblock.debug.$now”, (where “$now” is the absolute number of seconds elapsed). References to adblock in the the trace file will then be displayed in the web interface.

Viewing adblock log entries


Blockfile size limit (v2 only):

Adblock v1 may crash FreshTomato if it loads lists whose combined information exceeds your router's resource capacity. This can be difficult to troubleshoot. Adblock v2 comes with a customizable “limit” field. If the field is left empty, FreshTomato will calculate its value automatically. It will also automatically calculate the value if the router was reset to defaults.


20230125-184022.png


When you use external storage, the limit is calculated as 10% of physical RAM. When no external storage is found, the limit is calculated as 6.5% of RAM. The limit can be also manually configured. However, if your device becomes unstable, please revert to the auto-calculated value.

This is a limit, not a target. There's nothing wrong with having a smaller blockfile than the limit. A larger blockfile will result in longer restart times for the dnsmasq service. This is important because DHCP/DNS/TFTP won't function until dnsmasq finishes restarting.


Custom Path (v2):

An important v2 feature is the option to configure a path to permanent storage where Adblock can store relevant files. Doing this is strongly advised-it enables extra functionality of the script. It allows Adblock to store whole lists, their http headers, and the actual compiled blockfile there. This offloads RAM demand (/tmp) to permanent storage, and allows for information to survive a reboot. This makes troubleshooting easier.




It is also helpful when the script is rerun (manually or automatically) and there are no updated lists or configuration changes. The check for configuration changes involves controlling any modification to NVRAM variables since the last script run.

When a custom path is defined, Adblock will:

  1. Download all http headers from only the enabled lists.
  2. Compare those headers with the ones stored locally.
  3. Decide whether to re-process the blocklists or not.


This runs as follows:

  • If the configuration did not change and;
  • No updated lists are available;
  • Skip the re-processing and return to idle.


This saves time and resources.

“Configuration”, in this case, is any externally-mapped files where blacklist_custom or whitelists are defined (For example: /mnt/USB/my_whitelist.txt). Using a Custom path also allows adblock to load faster post-boot.


Domain Blacklist URLs & Group-of-Lists

This table contains a list of blacklists FreshTomato can download and use to block ads.

On: Checking this makes a gold star appear. The star confirms Adblock will enable downloading of the blacklist in that row. After selecting which blacklists you wish to use, click Save.


Blacklist URL: This shows the location on the Internet where that blacklist can be found.

Since release 2023.4, default settings don't include any blocklists, to reduce NVRAM demand. You can add lists manually. The official Adblock (DNS filtering) lists page summarizes them.



Description: Displays a name (if the publisher used one) or reference for the blacklist in that row.


Delete: If you move your mouse over a row with a blacklist, a red “x” appears at the end of the row. Clicking “x” deletes the row, and permanently deletes the blacklist.

There is no option to reset these to the original Blacklist URL entries.

If you delete a URL that's important to you, you will need to:

  • Re-enter it into the Blacklist URL table or;
  • Reset FreshTomato to default settings


However, for reference, the wiki contains a list of default blacklists: Adblock (DNS filtering) lists


Add: Inserts a blank row. In this row, you can type a new URL from which to download and use an additional Blacklist URL.

You can add a comment in the Description field.


Compatible data formats

Target lists to download must be in plain text. V2 can extract domains from lists in a variety of formats. Essentially, adblock uses a regex against each line in the file.

  1. Lines from the source file are parsed.
  2. Comments are removed.
  3. The domains are extracted.


This is true regardless of where (in which column) those domains are in the list.


Group-of-lists

Currently, v2 also accepts a new list format called “Group-of-lists”. This involves feeding the script a URL of links to a list of further URLs. The Group-of-lists is great at saving NVRAM space.


This can be visualized as follows:

URL |--> list


or

URL |--> Group-of-lists-file |--> URL |--> list
                             |--> URL |--> List
                             |--> URL |--> List
                             |--> URL |--> List



A limitation of the Group-of-lists format is that it doesn't verify the lists to assess whether they need to be downloaded or not.

When needed, adblock automatically handles EOL (End of Line) characters in files by converting them internally.

Group-of-lists are visible in the log. For example, if you see: “[8][2]” it would mean adblock is processing the 8th defined list in the Domain blacklist URLs & Group-of-lists on the second line (URL).

Domain blacklist custom


Domain Syntax

The syntax for this list is as follows:

  1. Standard domains (one entry per line).
  2. A “#” or “!” preceding text marks the line as comment only-to be ignored by script.
  3. A domain prefixed with “+” blacklists the domain and prunes subdomains found in the final blockfile.



This table summarizes the Domain blacklist syntax:

Adding custom entry Will blacklist / Do the following
sub.domain.com sub.domain.com
“sub2.sub.domain.com”
and all subdomains of “sub.domain.com”
+sub.domain.com sub.domain.com

Will also prune blockfile of dupes/redundant entries.
This reduces RAM usage, CPU cycles and storage space
# or ! Any string defined before them is considered but the rest of the line is ignored.


You can also specify a path to a file of domains (one domain per line). For example: “/mnt/usb/blacklist.txt”. Using external storage is both convenient and offers certain advantages for maintaining your blocklist. For example, moving a blacklist to another router would be as simple as removing a USB flash drive from one router and inserting it into the other (assuming both were set with the same storage location setting).


Sort a-z ↓ : introduced in release 2024.1, this sorts the contents of the field alphabetically.


Domain whitelist


Domain whitelist syntax

The Domain whitelist can use:

  • Standard domains (one entry per line).
  • Prepending a “%” sign before an entry indicates a “strict” whitelisting.
    Whitelisting always takes precedence over blacklisting.
    For example, if a blacklist contained “ads.example.com”, whitelisting
    “example.com” will undo the blacklisting of “ads.example.com”
    and all other subdomains.
  • Prepending a “#” or “!” before an entry marks it as comment only-
    it will be ignored by Adblock.


This table summarizes the Domain whitelist syntax:

Adding custom entry Will whitelist / Do the following
sub.domain.com sub.domain.com
sub2.sub.domain.com
and all other subdomains of sub.domain.com
%sub.domain.com sub.domain.com
but not subdomains of sub.domain.com.

Will also prune blockfile of dupes/redundant entries.
This reduces RAM usage, CPU cycles and storage space.
# or ! Any string defined before them is considered, but the rest of the line is ignored.


You can also specify a path to a file of domains (one domain per line). For example: “/mnt/usb/whitelist.txt”. Using external storage is both convenient and offers certain advantages for maintaining your blocklist. For example, moving a whitelist to another router would be as simple as removing a USB flash drive from one router and inserting it into the other (assuming both were set with the same storage location setting).


Sort a-z ↓ : Clicking this button sorts the field contents alphabetically.


Maintaining the Domain whitelist

One way to maintain the whitelist is to share it on your LAN, and teach users nslookup verification and whitelist additions.

Such a process could be automated as follows:

  1. Map a file in the Whitelist section of the Adblock menu
  2. Share the file/folder via Samba, in the File Sharing menu
  3. Teach users:
    1. How to use nslookup on a URL to verify DNS lookups are being poisoned
    2. Where to find the whitelist (via its samba share) and;
    3. After the whitelist has been edited, to configure settings in Buttons/LED as follows:




      With these settings, pressing the correct router button triggers the /usr/sbin/adblock update function.
      The time to complete an update varies. A swift run may take a few seconds. A full run might take minutes.



Enforcing Client Compliance

For Adblock to work properly, client devices must be configured with FreshTomato's IP address (dnsmasq) as their DNS server. This is done by configuring a static IP address on the client device itself or through FreshTomato's DHCP Service.

For the latter, enable DHCP. Then, enable the DNS server, and select “Use internal DNS” in the DHCP/DNS/TFTP menu.

These steps are mandatory. Clients that bypass FreshTomato's DNS server won't have their ads blocked.


Three optional settings improve proper Adblock operation:

  1. Enable Intercept DNS port in the DHCP/DNS/TFTP menu.
  2. Enable Prevent Client auto DoH in DHCP/DNS/TFTP.
  3. Enable the “DoH Server” list in Adblock to prevent DoH requests
    from being resolved. The default DoH Server list since 2021.6
    can be found at: oneoffdallas: DoH Server List


V1 functionality is a reduced version of scripts from outside FreshTomato. Don't enable both an outside script and FreshTomato Adblock at the same time. It may cause conflicts.

Adblock v2 uses advanced methods to block ads. It should be the preferred choice whenever possible.

Adblock/DNS Filtering affects name resolution only. Adblock cannot block an application that communicates directly via IP address.

Thorough management of domain blocking can be tedious. For example, with email spam, you'll probably have to deal with false positives.

Regardless of version, the script is always named “adblock”, and can be found at: “/usr/sbin/adblock”.


Adblock v2

Adblock v2 improves on the v1 script. It adds a richer feature set, increased user interaction (via toolbar) and command line options.







v2 Improvements


  1. Filters DNS similarly to v1, but in a more precise, controlled way.
  2. Lets you run only certain script functions to save time.
  3. Doesn't depend on list formats, as long as they're in plain text.
    It just extracts domains from the lists with different layouts.
  4. Supports Easylist format but this is not advised, as it may create
    false positives. For details, see: https://easylist.to/
  5. Can prevent resource issues. It assesses blocklist capacity before running.
    It can self-diagnose and self-heal some basic resource problems.
    e.g. dnsmasq not running due to lack of RAM.
  6. Supports Swift-run mode. You can add a domain to lists without having
    to reprocess (download) whole lists again.
  7. Supports external storage. Lists and headers can be stored there to
    help decide which lists to download, reducing traffic/processing time.
  8. Prevents false positives via a 30-minute interval between runs.
    This enables after updates.
  9. Includes more troubleshooting options. Debug logging and
    script tracing can look deep into the lowest operating level of the script.
  10. Can be operated via web interface or command line.



To get help with adblock at a command line, type adblock help :

adblock help output









Further debugging files can also be accessed at the command line by typing: adblock status .







To open the last trace file automatically, type adblock trace .

You'll be prompted to select the viewer in which to view trace files.






Using the v2 Controls


The Controls button bar in the Advanced section let you control adblock script execution.


Load: (start) will try to run the script. If the last run was a full one, and then it stopped, Load will exit the script and recover functionality (normal operation?) without a full processing of lists. This saves processing time. Load also occurs at boot time.

Unload: (stop) stops the script, regardless of what it's doing. It will also force-kill any child process if this is to be run while the script is already loading. If a full version of the script is found and external storage enabled, Unload will move the blockfile from external storage without destroying it. This allows very quick load times. Unload preserves the adblock.dnsmasq file.

Update: forces an update. There is a 30-minute interval between updates. Thus, if Update is run too often, adblock may not update, but instead, log why (active hold-time) it didn't update. When relevant, both GUI and command line will report the interval time. To force an update, click Unload, and then Update. This bypasses the interval period.

Reset-limit: nvram unset adblock_limit variable and recalculate the default value for you.

Clear all files: This is helpful when certain configuration problems occur, such as when files or NVRAM become corrupted, or adblock_limit or dnsmasq.adblock become too large. The former can happen if adblock_limit was calculated before considering new or larger running processes.

In such cases, if changing settings doesn't fix the problem, the “Clear all files” button will:
- Kill all adblock-related processes that are still running or stuck.
- Delete all existing adblock-related data files from pemanent storage.
- Delete all adblock-related data files from /etc and /tmp.
- Restart dnsmasq without adblock, (a last resort).

Running Unload, Reset Limit, Clear all files is a more radical approach to resetting things.
The biggest difference between clear and unload is that clear creates an environment as if adblock had never run, whereas Unload preserves adblock.dnsmasq. This makes quick-start possible when only unload has been run.
Snapshot: is for troubleshooting. It captures the list of relevant files and saves it to: /tmp/adblock.snapshot.${now.in.seconds}. You can take multiple screenshots before, after and when the script is running. A popup confirms a snapshot was taken and the file location.

Enable Only: In v1, to enable/disable adblock, you had to Save. This isn't always convenient, especially if a lot of lists are loaded. This lets you enable adblock (nvram set adblock_enable=1 + nvram commit) on the stand.

Disable Only: lets you disable adblock (nvram set adblock_enable=0 + nvram commit) without clicking Save. \\



Refresh: A new command at the bottom of the table. Refreshes are limited to a minimum frequency of 5 seconds. Refreshing more frequently will cause the system load to climb.

Swift-run (v2) and Full-run (v1) Operations




Modifications to the adblock script configuration are categorized as follows:

  1. List (enable/disable)
  2. List content update
  3. Change of parameters (loglevel / blockfile limit / path )
  4. Addition of a simple blacklist_custom
  5. Addition of a simple whitelist
  6. Addition of pruning blacklist_custom
  7. Addition of a strict whitelist
  8. Removal of a blacklist_custom
  9. Removal of a whitelist


The only two operations from the above list that can be run in Swift-run mode are:

  • No. 4 - Addition of simple custom blacklist domains
  • No. 5 - Addition of simple whitelisted domains


This is because they don't require full processing.

These are the most common operations performed when maintaining adblock script settings. Swift-run operations are performed live on a running blockfile. Therefore, the adblock script must be loaded in order to run functions (4) or (5). A swift run can be completed in seconds. A full run may take minutes. All operations other than (4) or (5) require a full run.

Please note that “swift” is a temporary action to allow your change to become immediately functional. In contrast, a successive full run (manual or scheduled) would process added domains in the standard way.

Command line operations (v2)


Another new v2 feature is the ability to interact with adblock at the command line.

As seen above in this “adblock help” screenshot, v2 has many more functions available:



The table below summarizes all command switches (v2 only):

Command switch Type the following Performs the following task/Comments.
Help “adblock help” Displays a summary guide of valid commands.
Status “adblock status” Gives constant updates on adblock's operational status.
Start “adblock start” The same as “Load” in the Web interface.
Stop “adblock stop” The same as “Unload” in the Web interface
Update “adblock update” The same as “Update” in the Web interface.
Upgrade “adblock upgrade” Checks online for script upgrades. If one is available,
it prompts you to install the upgrade. This downloads the latest script
version into /tmp (RAM) then mounts it over the built-in version.
Since the script is stored in RAM, it will be lost after a reboot.
Thus, an extra parameter is supported: “adblock upgrade silent”.
If an upgrade is available, this triggers it without prompting you.
This may be used in the init script, to ensure the latest version runs,
even after a reboot.
Test “adblock test $domain” Does a quick test to see if a domain name is being blocked.
Helpful when troubleshooting.
Reset “adblock reset” The same as “Reset” in the web interface.
Clear “adblock clear” The same as “Clear all files” in the web interface. Resets the blockfile maximum limit.
Trace/Status “adblock trace” Only available at the command line. When the script is run
with Loglevel set at 7,
adblock runs in “trace mode”.
Every command executed by the script is saved
into a trace (log/debug) file. Find the trace file
using “adblock status”. Running “adblock trace” will open the latest
appropriate trace file even more easily.
Snapshot “adblock snapshot” The same as “Snapshot” in the web interface.
Enable “adblock enable” The same as “Enable” in the web interface.
Disable “adblock disable” The same as “Disable” in the web interface.



Adblock test command (v2)


For quick and easy testing of broken resolution or other DNS issues, use the “test” switch.

Typing: “adblock test example.com” will quickly test whether name resolution is functioning.




Specifically:

  • Dnsmasq answer - Displays the actual name resolution as seen
    by the router and LAN users.
    The word “Blocked” is displayed if access to the domain was blocked.
  • Cloudflare answer - This query bypasses the usual dnsmasq process
    and sends a query to address 1.1.1.1 .
    This can help verify if a domain exists.
  • Blockfile ref - Checks the blockfile for references to the tested domain.


With these 3 pieces of information, it's easy to assess whether a domain is blocked, non-existent, or intact.

Automatic list updates

Normally, adblock adds a crontable entry to perform a daily list update. Update times are now randomized to prevent DDoS of the list providers' servers. Each time you run adblock stop or reboot, a new update time is set. The update's time window varies between 02:00 AM and 05:59 AM local time, depending on the script version run (original v1, modified v1, or v2).


To verify the current scheduled update time, type:

root@router:/# cru l | grep adblockJob
28 4 * * * /usr/sbin/adblock update #adblockJob#


In this example, it's 28 (min) 4 (am) * (Every day) * (Every month) * (Every day of the week).

This picture illustrates the syntax of crontable expressions:



For help automating/verifying correct crontable expressions, see: crontab.guru

If you stop adblock, the crontable expression is removed. This is by design.

Future development goals

  • Add an option to let FreshTomato users share their Whitelists.
    This would allow creation of a shared database of “good” domains,
    based on user experience.
    This shared whitelist would be downloaded the same way
    you would download any blacklist.
  • Add a test function to the web interface (currently only via CLI).


Adblock Notes and Troubleshooting

Troubleshooting v1

You can test Blacklisted or Custom entries by using the nslookup tool (Windows/Linux/MacOS). Simply run nslookup and enter the domain/URL to be tested. If it resolves to 0.0.0.0, then Adblock is working properly, and the domain is being blocked. You can also check FreshTomato's System log (Syslog) to see if Adblock activity is reflected there.


For example, this entry shows that dnsmasq replaced the true IP address of hbx.media.net with 0.0.0.0.

Jan 9 19:57:29 rt-n66 daemon.info dnsmasq[4872]: config hbx.media.net is 0.0.0.0

Adblock worked properly.


Crashing

If your router crashes, you may be using too many large Blacklists, which can exhaust available RAM. Try using smaller blacklists, or fewer large ones. Adblock v2 is much less likely to exhaust RAM.


Websites for Testing

Some websites allow you to test the effectiveness of your Adblock configuration. When using them, ensure you don't have adblock/script-block plugins enabled in your browser or your results will be invalid.

Some good adblock testing sites include:


Adblock won't work with DoH, DoT and other Encrypted DNS lookups

Increasingly, devices use DNSSEC, and encrypted DNS protocols, such as DoH [DNS over HTTP(S)]. Adblock won't work with these lookup methods. This is because if lookup requests are encrypted or directed to a third-party server, Adblock can't intercept, and therefore can't poison them.

Enabling the “DoH Server” blacklist in the Blacklist URL table may help with this. It will download, then block access to common DoH servers.


Troubleshooting v2

When troubleshooting, it's best to set the loglevel at 7, (as described in the Settings section above). Then, analyze the logs and trace file. Often the list publishers suffer from issues such as downtime, or a broken cache. It's common to see adblock fail to download a list one day but download it the next day without issue.


In order to learn what might be wrong, you should understand adblock's status indicators.

Here are a few signs which may indicate a problem:

  • System load is too high in the 5min+ section (For example, > 1.5)
  • RAM usage is too high (> 90%)
  • Dnsmasq ownership remains root (non-operational) until lists are fully loaded.
    It's normal to see “root” briefly, but this should switch to “nobody” within 20 seconds.
  • Too many dnsmasq restarts
  • Dnsmasq restart time is too high (> 15 seconds)
  • Adblock calls today
  • Last run errors


If the Blockfile size exceeds the Blockfile size limit value, the file size will be reduced. If (with limit manually set high) dnsmasq still doesn't run properly afterwards, the safeDnsmasqRestart() process will reduce the Blockfile size in increments of 5%. After each reduction, it will reattempt to fully start dnsmasq. When dnsmasq functions properly, the new auto-calculated limit is set as the current Blockfile size limit.


When a trace is running, FreshTomato will display this (FIX ME!) in the Web interface:




The additional debugging files can also be accessed at the command line by typing: adblock status






To open the last trace file automatically, type adblock trace at a command shell. You'll be prompted to select the viewer in which to view trace files.







Apple iOS devices (iCloud paid service only)

iOS devices have settings which may interfere with Adblock operation. “Limit Address Tracking”
can be disabled on a per-WLAN basis, seen here:








Alternatively, you can disable it globally via the “Private Relay” setting, seen below:





You might connect your mobile/laptop to different WiFi networks at different locations. For this reason, it makes sense to disable Private Relay on a per-WLAN basis. It is wise to disable it wherever FreshTomato provides your Internet connection and Adblock is enabled.


Remember, these options are only available to paid iCloud customers.



1)
You can adjust settings for both caches in the dnsmasq custom configuration field in the DHCP/DNS/TFTP menu.
advanced-adblock.txt · Last modified: 2024/10/05 22:07 by hogwild