Wildcard and regex support for whitelisting

Feature Requests Implemented
1

hello developers!

I'd like the ability to use wildcard (and regex) on the whitelist.

The reason being, our office is pretty much "all in" on Microsoft 365, Sharepoint, etc. Basically, "all microsoft, all the time". Yet, several of the blacklists I subscribe to block various microsoft servers, causing random issues, like logging in, etc. While I understand this may not be everyone's cup of tea, it is what it is for us and it would be great if we could wildcard-whitelist .microsoft.com once and for all.

Conversely, we're mainly Mac users at home. And I see on some blocklists things like ocsp.apple.com listed, as well as other servers. I have no clue why anyone would think it would be a good idea to block a ocsp server! But same thing, at home I'd like to whitelist everything .apple.com.

-Rick

2

In the short term, you could reduce your false positives with fewer block lists. One of the problems with public block lists is that you have no control over the content, and what the list maintainer wants to block may not be what you want to block (as in the case of ocsp.apple.com, which I have also had to whitelist).

You might try just a few lists (or no lists) and set up some regex filters that you control. Here are some examples that will knock down a lot of the adware, metrics, etc.

^(.+[-_.])??ad[sxv]?[0-9]*[-_.]
^adim(age|g)s?[0-9]*[-_.]
^adse?rv(e(rs?)?|ices?)?[0-9]*[-.]
^adtrack(er|ing)?[0-9]*[-.]
^advert(s|is(ing|ements?))?[0-9]*[-_.]
^aff(iliat(es?|ion))?[-.]
^analytics?[-.]
^banners?[-.]
^beacons?[0-9]*[-.]
^clicks?[-.]
^count(ers?)?[0-9]*[-.]
^pixels?[-.]
^stat(s|istics)?[0-9]*[-.]
^telemetry[-.]
^track(ers?|ing)?[0-9]*[-.]
^traff(ic)?[-.]
2 Likes
3

Hey jfb,

Just a quick shout-out thanks for this list of regex filters! I've never really been able to wrap my head around regex, so this list is great! Thanks much!

Someone should consider adding this list as "examples" on the regex section of the documentation! :wink:
https://docs.pi-hole.net/ftldns/regex/tutorial/

-Rick

1 Like
4

I won't take credit for these. They are on github, but for the life of me I can't recall where. When I find the link, I'll post it.

Edit - thanks @anon55913113, that was the link.

6

regexp whitelistling would be a really nice feature because now it is possible to set .* as blacklist regexp and deny everything but it is really cumbersome to add all subdomains that are to be allowed instead of just adding *.microsoft.com for an example.

8

I'm not sure i'm following what your saying but the webgui has 'exact', 'wildcard'(which also is regexp), 'regexp' for blacklisting. But only 'exact' for whitelisting. What I would like and I'm guessing this feature request is, would be to have the regexp for whitelisting as well.

Thanks for a fantastic software for any developer who's reading.

10

ah ok, that's nice. Care to share? Or even better submit to the project?

1 Like
12

I would (still) welcome wildcard whitelisting incorporated into pi-hole.

We are a Microsoft 365 Business shop (for better or worse), and literally just last week I had to whitelist yet-another microsoft sub-domain after my users couldn't login into Skype for Business because some blocklist decided it needed blocking. Grrrr....

1 Like
13

@Rick_v,

For what its worth, Microsoft publishes domain and ip addresses at https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges including machine readable json linked from there.

2 Likes
14

@smoser,

Wow, this is tremendously helpful! Thank you so much!

15

Me too. It very much surprised me it wasn't there, while pihole allowed me to put
*.somedomain.sometld
in whitelist.txt, it didn't error out or anything.

16

Is there an easy way to reload dnsmasq from this script from outside of a docker-container? I mean, I would have to reload the list manually everytime the list updates, that's not going to work.
Also, it would be nice if pi-hole had an 'execute after' option for such scripts, like you write, a hook during import or something post all updates, just before reload.

18

2 posts were split to a new topic: Entering multiple regex at one time

19

Revisiting the original request: Whitelist regex support.

It is technically possible, however, I will tell you why I don't think it is a good idea:

Regex filter evaluation is - always - a sequential (and hence slow) task. You have to try all of them until you know that none of them matched. This is the exact reason for why we split the blacklist into an "exact" and a "regex" component. The "exact" component is loaded into cache and can be replied to with close to no delay at all. Walking the chain of regex filters is, however, much slower.

The implementation could be made in two ways:

  1. Only use regex-based whitelist - very bad performance if you have many whitelisted domains
    This is to be avoided as Pi-hole v5.0 will just introduce support for massive whitelists, using an implementation strategy that will still give the result of a query with a typical delay of < 4 msec even if your have millions of domains on the whitelist.
  2. Add a regex-based whitelist next to the already existing whitelist - increase in complexity for the users.
    This is to be avoided as well as it would introduce a severe slowdown of the blocked domains preparation (AKA "gravity"). Instead of only excluding the whitelisted domains (which is very efficient), we'd need to evaluate all whitelist regex filters against any of the (possibly up to millions of) domains on the blocking lists. This would result in a catastrophic slowdown, maybe causing gravity to take hours instead of tens of seconds on Raspberry Pi devices. This is unacceptable.
1 Like
21

As you know there is almost nothing I will not discuss about. If I can be convinced from the contrary, I have not problems in accepting that I have maybe been wrong.

Can you give some more details about this so we can understand the performance impact?

  1. What kind of device are you running Pi-hole on?
  2. How many blocked domains so you source without whitelist regex filtering?
  3. How many whitelist regex entries do you use?
  4. How complex are they (this is a asking for subjective feeling, .* is not very complex, however, if you use () or | rather often, then the regex is much more complex when compiled to byte code).
  5. How many blocked domains are left after your whitelist procedure is done with them?
  6. How long does the script take only for the whitelist processing?
23

Hmm, okay I will continue to think about this, however, we made recent optimizations for v5.0 which make implementing it in the way you did (a bit) harder.

As far as I see, your implementation also only cleans the gravity list, right? If a user (intentionally or not) blocked something in addition on the blacklist, you don't delete the line for him, right? If so, this would be rather unexpected behavior I'd say.

Instead of one big gravity run, that needs to be repeated for each whitelist modification, we instead use table views now. By this it is sufficient to send SIGHUP to pihole-FTL for the modified whitelist to become active. There is no call to pihole -g any more. However, this also means that edits to such a regex-based whitelist would again require us to run gravity each time as live filtering when loading the lists may be too slow.

Can you send me the regex filters you're using so I can have a realistic set of filters for performance measurements when I come around doing a testing implementation?
Please do not see this sentence as a guarantee that I will do it anytime soon nor that this will become part of Pi-hole if it turns out to be either too slow or to complex.

25

Quicker than expected, we now already have a development draft that is handling regex whitelists consistently and completely inside of FTL and are currently working out the performance impact.

I've seen the blacklist regex domains on @mmotti's GitHub project but I'm not sure I have seen the mentioned whitelist regex before.
Note that this feature request is about whitelisting, not trimming down the number if gravity domains because they are already partially covered by regex filters.

27

A post was split to a new topic: DNS configuration for wildcard whitelisting

30

How about the Pi-hole Teleporter feature? Note that it is not only available on the dashboard but also through the CLI ( pihole -a -t creates a pi-hole-teleporter file in the current working directory ).

32

I agree.

1 Like