# Gravity.list and regex

**URL:** <https://discourse.pi-hole.net/t/gravity-list-and-regex/15448>\
**Category:** General\
**Created:** [December 25, 2018, 12:37pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448 "2018-12-25T12:37:07Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jpgpi250](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jpgpi250/32/286_2.png) [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)\
**Post date:** [December 25, 2018, 12:37pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/1 "2018-12-25T12:37:07Z")

</div>

Informational only...  
I was interested in the effect of using regular expressions.  
For this test, I reconfigured pihole to using the default lists only:

```auto
StevenBlack https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
MalwareDom https://mirror1.malwaredomains.com/files/justdomains
Cameleon http://sysctl.org/cameleon/hosts
ZeusTracker https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
DisconTrack https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
DisconAd https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
HostsFile https://hosts-file.net/ad_servers.txt

```

I'm using the [regex list](https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list) from @mmotti

Running this script (does take a while - there probably is a better way to achieve this - I said before, NOT a Linux expert, nor a regex expert…):

```auto
#!/bin/bash

regexfile=/etc/pihole/regex.list
gravityfile=/etc/pihole/gravity.list

list=0
regexcount=0
result=0

for domain in `cat $gravityfile`
do
   list=$((list+1))
   valid=true
   for regex in `cat $regexfile`
      do
         if [[$domain =~ $regex]]; then
            regexcount=$((regexcount+1))
            valid=false
            break
         fi
      done
      if $valid; then
         result=$((result+1))
      fi
done
echo -e "listsize=$list - regex matches=$regexcount - unique domains=$result"

```

The result:  
number of entries in the list: **136702**  
number of regex matches: **25342** ( **18.5%** )  
number of domains, not covered by regex: **111360** ( **81,5%** )

There are several topics on improving the `gravity.list`, based on the content of `regex.list` ([here](https://discourse.pi-hole.net/t/cleaning-gravity-list-based-on-what-is-in-regex-list/12009), [here](https://discourse.pi-hole.net/t/regex-the-next-level/12707) and [here](https://discourse.pi-hole.net/t/using-regex-for-filtering/10959) for example...). These are ingenious, but complex procedures, witch take a lot of processing time and make things complex.  
It should however, given an improved and much faster version of the script above, be possible to integrate this in `gravity.sh` (= `pihole -g`). Is it worth the effort?

discuss...

---

<div class="post-metadata">

**Author:** ![Mcat12](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/mcat12/32/20_2.png) [@Mcat12](https://discourse.pi-hole.net/u/Mcat12)\
**Post date:** [December 28, 2018, 6:50pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/2 "2018-12-28T18:50:54Z")

</div>

It is more efficient to block domains using the normal method instead of regex. The domains are kept in a structure which allows constant time lookup (very fast). When using regex, the domains have to be checked (in the worst case) against every regex you have loaded, and only after all of that does it know if it is blocked or not (this result is kept until the regex is modified).

---

<div class="post-metadata">

**Author:** ![jpgpi250](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jpgpi250/32/286_2.png) [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)\
**Post date:** [December 28, 2018, 8:07pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/3 "2018-12-28T20:07:03Z")

</div>

> [@DL6ER](#):
>
> For **already known** domains there is no notable difference as regex validation is not happening and we directly lookup the domain in the cache. If not found, we lookup the configuration and finally sent upstream if nothing prevents us from doing this.

As with unbound, there will be a delay the first time the domain is queried, after this, the answer is readily available.  
I understand there is a pro and con side to this approach, but, for example, I currently have **13949** (no typo it's really **13949** , as counted by notepad++ / find / count) matches for `doubleclick.net` in my gravity list. **I'm not sure witch method will be the fastest to block the domain (wildcard, regex or gravity)**.  
Other examples:  
`centade.com`: 8397  
`clickbank.net`: 631

I have, default output from `pihole -g`, no tweaks, 1280858 gravity entries, 15 regex expressions and 106 wildcard (will be 53 - IPv4 and IPv6, as soon as dnsmasq2.80 is integrated in pihole-FTL) entries.

---

<div class="post-metadata">

**Author:** ![DL6ER](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/dl6er/32/281_2.png) [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)\
**Post date:** [December 28, 2018, 8:20pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/4 "2018-12-28T20:20:21Z")

</div>

> [@jpgpi250](#):
>
> I currently have **13949** (no typo it’s really **13949** , as counted by notepad++ / find / count) matches for `doubleclick.net` in my gravity list.

But many of these are things like

```auto
www.ad.doubleclick.net.76530.9544.302br.net

```

which rarely exist (admittedly, this is only a feeling).

> [@jpgpi250](#):
>
> **I’m not sure witch method will be the fastest to block the domain (wildcard, regex or gravity)**.

The fastest will be to have everything in gravity as the `dnsmasq` cache is optimize for fast lookups of domain in cache buckets. This is much faster than wildcard or regex. Wildcards are looped over and it is checked for each requested domain if this domain is part of the wildcard. With regex it is much more dramatic as every regex evaluation is a complex process - with all the possible rules obviously much slower than a simple sub-string comparison. The high regex performance that is seen in FTL is only possible as we pre-compile regex and hence make them as fast as possible in their execution.

TL;DR: gravity.list can as grow as it can fit into memory. The only slowdown you'll observe is the initial loading. Both, wildcards and regex filters do not have any initial overhead, however, their evaluation is notable or even significantly slower than a direct domain cache hit from domains imported from gravity.list.

---

<div class="post-metadata">

**Author:** ![jpgpi250](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jpgpi250/32/286_2.png) [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)\
**Post date:** [December 28, 2018, 8:30pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/5 "2018-12-28T20:30:01Z")

</div>

> [@DL6ER](#):
>
> With regex it is much more dramatic as every regex evaluation is a complex process

I still don't understand, given the above statement, why you converted the wildcards into regex. I'm still using wildcard, just changed the filename to avoid `regexconverter="/opt/pihole/wildcard_regex_converter.sh"`

> [@DL6ER](#):
>
> The only slowdown you’ll observe is the initial loading

That's why I feel cleaning up the gravity list would be a benefit. Doesn't pihole-FTL needs to load the list, every time you give it a kick (SIGHUP)? wouldn't a cleaned gravity list benefit the load time.

---

<div class="post-metadata">

**Author:** ![jpgpi250](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jpgpi250/32/286_2.png) [@jpgpi250](https://discourse.pi-hole.net/u/jpgpi250)\
**Post date:** [December 28, 2018, 11:05pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/7 "2018-12-28T23:05:44Z")

</div>

> [@anon55913113](#):
>
> My understanding is that from 20 subdomain entries in Gravity then it would be effecient to switch over to Regex for that specific domain.

Not so sure about that. from what I understand from @DL6ERs explanation, if your regex list is longer, it will take more time to do the evaluation. The same logic would apply to wildcards, the more you have, the longer it takes to process them.

I assume wildcards are read from file, by pihole-FTL, once, and kept in memory, since the dnsmasq file clearly states you need to stop/start dnsmasq for changes to apply.  
I assume the same goes for regex, @DL6ER: could you please confirm/deny this?

---

<div class="post-metadata">

**Author:** ![DL6ER](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/dl6er/32/281_2.png) [@DL6ER](https://discourse.pi-hole.net/u/DL6ER)\
**Post date:** [December 29, 2018, 12:33pm UTC](https://discourse.pi-hole.net/t/gravity-list-and-regex/15448/8 "2018-12-29T12:33:24Z")

</div>

> [@jpgpi250](#):
>
> if your regex list is longer, it will take more time to do the evaluation. The same logic would apply to wildcards, the more you have, the longer it takes to process them.

Yes.

> [@jpgpi250](#):
>
> I assume the same goes for regex, @DL6ER: could you please confirm/deny this?

Yes. Wildcards need restarting. Regex can also be updated on-the-fly by sending the signal `SIGHUP` to `pihole-FTL`. The difference is that `dnsmasq` is constructed such that configuration can only be read at startup while `pihole-FTL` can re-read its config whenever needed and regex is an FTL implementation, there is _no_ regex code at all in the contained `dnsmasq`.