Adlist vs regex blacklist

The issue I am facing:
I know this is 'expected behavior' but I'm running into a little bit of frustration with the fact that adlists take precedent over regex blacklist rules when blocking happens.
Basically, there's some domains that I want to set the ";reply=" option on, but these domains are already being blocked by various adlists I use. So now I have to either give up on doing this, or throw the baby out with the bathwater and stop using 5 entire lists just to avoid adlisting like 2 domains just so my blacklist regex will kick in.

Is there some option somewhere to make blacklist regex rules take precedent over adlists, or else make gravity omit certain domains when importing from adlists?

My current solution involves setting some rather counter-intuitive groups up and omitting those 5 adlists from applying to a few devices on my network that regularly hit those domains.
Again, however, this means all of the rest of the protection from those lists are now gone for those devices, and I'm not a fan of that fact.

Details about my system:
I have a standard pihole install on a raspberry pi 3B. I use pihole-updatelists and I have pivpn installed, but nothing that should modify pihole itself.

What I have changed since installing Pi-hole:
Installed adlists, blacklists, one whitelist entry, and set it up as a DHCP server. Standard stuff.

Not a solution, but did you check how many unique domains those blocklists would actually add to your existing pool?
A third-party utility like yubiuser's adlist tool could provide some insights into your actual blocklist utilisation and help you to better understand how much you'd really lose by dropping a specific blocklist (and in general, may help you decide which blocklists to keep).

As to potential workarounds to your issue:
Could you help us understand your intention in blocking domains that way?
What is the actual reply your are trying to enforce?
What's the reason for suppyling a specific reply for those regex blocks?

Thanks! I'll give the adlist tool a try. I was also considering just writing a python script that would cobble together my own adlist from a list of adlists, using my own rules for excluding certain items.
The obvious problem with that, is that I would loose the ability to use pihole's built-in tool for determining which adlist is blocking a given domain.
(That, and I was hoping to write a guide for doing this that didn't involve installing/configuring third party scripts and tools.)

Generally speaking, the default blocking mode ("NULL") is the best response for most devices. However, some devices insist on getting spammy when blocked that way, so I've been experimenting with other blocking modes.
The problem with enforcing a global blocking mode, as the documentation suggests, is that yet more devices get even more spammy with some of them. ("NXDOMAIN" mostly)

The two URLs I've personally been having the most trouble with are api2.branch.io & scribe.logs.roku.com. The roku I'm less concerned with, since it only browses a certain set of sites anyway. (so putting it in a group and giving it a more limited subset of adlists is less of a concern) But the api2.branch.io link is used by a couple of our mobile phones and our phones start getting really chatty when they can't connect to that. (and, obviously, I want as many adlists as possible supporting our mobile devices.)
I've also heard that fitbit devices can cause similar behavior, though I don't own any to test with myself.

So far, I've got the best results out of pointing these things at a nonsense ip address they can time out on. (;reply=192.168.192.168) But I'm experimenting with the other reply options to see if any of them give better results. (which obviously takes some time, so I won't have a straight answer on this for a while)

So you are trying to answer requests for the exact api2.branch.io domain with an unassigned private IP, right?

In that case, I can think of another approach that may better match with your requirements:
a) allow api2.branch.io via Group Management | Domain management
b) create a Local DNS record with your chosen nonsense IP for it

1 Like

I actually started off doing that exact thing, and I guess I'll go back to that if these other reply types don't turn out better than that. I switched to doing this because I wanted to test other blocking mode options without upsetting other devices on my network. It took me almost a whole week to figure out it wasn't doing anything at all and why.
(by 'better' ofc I mean they reduce the spammy nature of these devices more)

Also, the (admittedly small) problem with that solution is that it does skew your stats a little, since it'll show as "allowed" even though you know you're blocking it.

What's funny to me is that there's entire dedicated FTLDNS configuration settings for mozilla canary and icloud blocking that makes specifically those domains get an NXDOMAIN response, regardless of your adlists or anything else. Clearly, adding hardcoded options like this isn't sustainable in the long run.

Instead, ideally, I would love to see a feature that just lets us specify things like blocking mode and maybe TTL on normal (non-regex) blacklist entries. Then we could block mozilla, icloud, roku, or any other thing with the appropriate response, and tell those devices to shut up for however long we want. I'm sure there's probably already feature requests like that out there already, but this regex 'solution' exists, so I'm trying to make it work.

Update: I'm getting REALLY good results from using logs\.roku\.com;reply=refused as a blacklist rule. I also have FTLDNS configured with BLOCK_TTL=60 (default is normally 2) so that might be playing a role.
Regardless, I'm going to try NXDOMAIN next to see what happens.

At any rate I'm going to assume, based on the lack of other suggestions for how to handle this, that I'm going to have to arrange my own solution to applying these rules around my adlists using some scripts or something.

I really wish there was a priority setting or something we could set on regex blacklist rules to make them override our adlists when needed. I understand why they normally don't, but that also detracts from their usefulness IMO.

You can vote here:

Thanks! I was actually just searching around to see if there was/n't already a feature request like this.

It's got my vote, obviously. :smile:

Have you tried whitelisting the domains in question, the domains for which you want to apply special handling? This would not require any changes to your adlists, and you already know the domains that will get special handling.