# Optimisation | Scripts for advanced users

**URL:** <https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567>\
**Category:** Customizing Pi-hole\
**Created:** [June 8, 2019, 1:31am UTC](https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567 "2019-06-08T01:31:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmotti](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/mmotti/32/9927_2.png) [@mmotti](https://discourse.pi-hole.net/u/mmotti)\
**Post date:** [June 8, 2019, 1:31am UTC](https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567/1 "2019-06-08T01:31:03Z")

</div>

**These are now all compatible with the Pi-hole DB (currently active on the dev branch)**

Selection of regexps with easy one-command install:  
[https://github.com/mmotti/pihole-regex](https://github.com/mmotti/pihole-regex)

**Example Output** :

```auto
pi@raspberrypi:~ $ curl -sSl https://raw.githubusercontent.com/mmotti/pihole-regex/master/install.sh | bash
[i] Fetching mmotti's regexps
[i] Fetching existing regexps
[i] No regexps currently exist in the database
[i] Generating CSV file
[i] Importing CSV to DB
[i] Regex import complete
[i] Refreshing Pi-hole

These are your current regexps:
  ^(.+[-_.])??adse?rv(er?|ice)?s?[0-9]*[-.]
  ^(.+[-_.])??m?ad[sxv]?[0-9]*[-_.]
  ^(.+[-_.])??xn--
  ^adim(age|g)s?[0-9]*[-_.]
  ^adtrack(er|ing)?[0-9]*[-.]
  ^advert(s|is(ing|ements?))?[0-9]*[-_.]
  ^aff(iliat(es?|ion))?[-.]
  ^analytics?[-.]
  ^banners?[-.]
  ^beacons?[0-9]*[-.]
  ^count(ers?)?[0-9]*[-.]
  ^pixels?[-.]
  ^stat(s|istics)?[0-9]*[-.]
  ^telemetry[-.]
  ^track(ers?|ing)?[0-9]*[-.]
  ^traff(ic)?[-.]

```

Optimise your gravity list/table by removing unnecessary entries (through your custom DNSMASQ wildcards and regexps)

> **[GitHub - mmotti/pihole-gravity-optimise: Optimise gravity.list with the help of wildcards /...](https://github.com/mmotti/pihole-gravity-optimise)**
>
> Optimise gravity.list with the help of wildcards / regexps

**Example Output** :

```auto
pi@raspberrypi:~ $ gravityOptimise.sh
[i] Pi-hole DB detected
[i] Updating gravity
[i] Parsing existing wildcard config (DNSMASQ)
[i] Removing wildcard matches from gravity
[i] Removing regex matches from gravity
[i] Updating gravity database
[i] Removing temp files
[i] Refreshing Pihole
[i] 55205 domains were removed from gravity
[i] 58103 domains remain in gravity

```

Add wildcard domains from adguarddns, easylist, easyprivacy and nocoin ([justdomains repo](https://github.com/justdomains/blocklists)) without conflicting with your whitelist or other local dnsmasq wildcards. Best when complimented by [gravityOptimise](https://github.com/mmotti/pihole-gravity-optimise).  
[https://github.com/mmotti/pihole-filter-lists](https://github.com/mmotti/pihole-filter-lists)

**Example Output** :

```auto
pi@raspberrypi:~ $ fetchFilterLists.sh
[i] Pi-hole DB detected
[i] Selected filter sources: adguarddns,easylist,easyprivacy,nocoin
[i] Fetching domains
[i] Parsing existing wildcard config (DNSMASQ)
[i] Cleaning domains
[i] Removing regex.list conflicts
[i] Checking for local wildcard conflicts
[i] Checking whitelist conflicts
[i] Determining output format
[i] Constructing output
[i] 31438 domains added to /etc/dnsmasq.d/filter_lists.conf
[i] Restarting FTL
[i] Done

Don't forget to run this script again if you make changes to your whitelist!

```

Identify gravity domains with \>=20 subdomains and add those as wildcards to a custom DNSMASQ conf file. Best when complimented by [gravityOptimise](https://github.com/mmotti/pihole-gravity-optimise)

> **[GitHub - mmotti/pihole-gravity-wildcards: Generate wildcards from gravity domains with...](https://github.com/mmotti/pihole-gravity-wildcards)**
>
> Generate wildcards from gravity domains with excessive subdomains

**Example Output** :

```auto
pi@raspberrypi:~ $ generateGravityWildcards.sh
[i] Pi-hole DB detected
[i] Updating gravity
[i] Fetching domains from gravity table
[i] Identifying domains with >= 20 subdomains
[i] 139 domains found
[i] Fetching domains from whitelist table
[i] Checking for whitelist conflicts
[i] 139 domains remain after conflict resolution
[i] Determining blocking mode
[i] Outputting domains to: /etc/dnsmasq.d/mmotti_generated_wildcards.conf
[i] Restarting Pi-hole service

```

---

<div class="post-metadata">

**Author:** ![jfb](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jfb/32/4332_2.png) [@jfb](https://discourse.pi-hole.net/u/jfb)\
**Post date:** [June 8, 2019, 1:53am UTC](https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567/2 "2019-06-08T01:53:26Z")

</div>

As previously noted by the developers, the gravity list is quite fast and regex filters are relatvely slow in comparison. What benefit does moving domains from the fast lane to the slow lane provide?

---

<div class="post-metadata">

**Author:** ![jfb](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/jfb/32/4332_2.png) [@jfb](https://discourse.pi-hole.net/u/jfb)\
**Post date:** [June 8, 2019, 2:01pm UTC](https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567/4 "2019-06-08T14:01:11Z")

</div>

> [@anon55913113](#):
>
> The break even point is, one regex must replace twenty or more gravity lines.

How have you determined that this is the break-even point?

> [@anon55913113](#):
>
> An important bit of information is in the first line. Mmotti is implementing this in the development version of Pi-hole which has no gravity.list anymore

Regardless of where the gravity list is located, it is still used as the master block list and is still searched for blocked domains.

---

<div class="post-metadata">

**Author:** ![mmotti](https://discourse-cdn.pi-hole.net/user_avatar/discourse.pi-hole.net/mmotti/32/9927_2.png) [@mmotti](https://discourse.pi-hole.net/u/mmotti)\
**Post date:** [June 8, 2019, 3:10pm UTC](https://discourse.pi-hole.net/t/optimisation-scripts-for-advanced-users/20567/7 "2019-06-08T15:10:13Z")

</div>

> [@jfb](#):
>
> As previously noted by the developers, the gravity list is quite fast and regex filters are relatvely slow in comparison. What benefit does moving domains from the fast lane to the slow lane provide?

With reference specifically to regex filters, there are several reasons why one may wish to use these. The main reasons being coverage and flexibility. A single regular expression can capture many, many different variations of domains; some of which may not even be covered by your gravity list.

**Example:** `^(.+[-_.])??m?ad[sxv]?[0-9]*[-_.]` accommodates for 19,318 `gravity` entries using the default host files.

```auto
pi@raspberrypi:~ $ grep -E '^(.+[-_.])??m?ad[sxv]?[0-9]*[-_.]' /etc/pihole/gravity.list | wc -l
19318

```

There are only currently 16 regexps in my filter list, and I would agree that if this would be expanded to hundreds / thousands, then this could have potential for moving things to the "slow lane".

On the subject of slow lane, that's why I have created some of these scripts. The current implementation of Pi-hole uses regexps for wildcards now which one could argue is a vastly more "expensive" than a simple DNSMASQ wildcard. That being said, of course I completely understand and appreciate the reasoning behind moving in this direction. I personally try to reserve regexp for use only when absolutely necessary (i.e. for more complex tasks / catchment).

DNSMASQ wildcards are particularly effective:

```auto
pi@raspberrypi:~ $ grep -E '302br\.net$' /etc/pihole/gravity.list | wc -l
17944
pi@raspberrypi:~ $ grep -E '2o7\.net$' /etc/pihole/gravity.list | wc -l
1474
pi@raspberrypi:~ $ grep -E 'adnxs\.com$' /etc/pihole/gravity.list | wc -l
560
pi@raspberrypi:~ $ grep -E 'doubleclick\.net$' /etc/pihole/gravity.list | wc -l
246

```

The "optimisation" script whos purpose is to remove gravity entries based on regex and DNSMASQ wildcards - One could argue that this is unnecessary as there is little impact on speed. This really comes down to personal preference again - I prefer to only have the absolutely necessary entries.