Pi-hole becomes unresponsive during pihole -d and -q

General Beta 5.0
21

I'm not sure why pihole -q is looking up localhost, nor why it is being called via lighttpd and php during pihole -d, I'm sure that's it's own problem that this may have brought to light...

Rather than do a full pihole -d (takes about 30 minutes), I ran time pihole -q google.com (only takes 11 minutes) after disabling conditional forwarding. It has not improved performance, and I'm still having the same issue. Here's a screenshot:

Imgur

The CPU usage fluctuates during the process, but RAM stays completely throttled the entire time.

22

That code hasn't changed in 6 months or so, but let's pull it apart.

sudo bash -v -x /opt/pihole/query.sh google.com and be prepared for a whole lot of output.

24

I'm going in! Wish me luck!

Will DM results, as they are likely to be long and probably contain some extra data on my network setup.

Initial result - it seems to be getting hung up while or just after going through my regex list. Will try removing all regex and see if that fixes it.

25

Alright, it was definitely my regex list. Posting here in case something obviously is wrong:

(^|\.)com\.private-domain\.tld$
(^|\.)net\.private-domain\.tld$
(\.|^)*\.metric\.gstatic\.com$
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?telemetry[_.-]
^(www[0-9]*\.)?xn--
^adim(age|g)s?[0-9]*[_.-]
^adtrack(er|ing)?[0-9]*[_.-]
^advert(s|is(ing|ements?))?[0-9]*[_.-]
^aff(iliat(es?|ion))?[_.-]
^analytics?[_.-]
^banners?[_.-]
^beacons?[0-9]*[_.-]
^count(ers?)?[0-9]*[_.-]
^mads\.
^pixels?[-.]
^stat(s|istics)?[0-9]*[_.-]
^track(ers?|ing)?[0-9]*[_.-]
^traff(ic)?[-.]

The majority of these came from: https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list except:

(^|\.)com\.private-domain\.tld$
(^|\.)net\.private-domain\.tld$
(\.|^)*\.metric\.gstatic\.com$

were built using the wildcard feature through pi-hole

^traff(ic)?[-.] was found on Reddit

Next step is to add these back in one-at-a-time and see which is the problem child.

1 Like
26

Found the culprit:

(\.|^)*\.metric\.gstatic\.com$

The * appears to break -q.

27

This is not a valid regular expression. The interpretation of (\.|^)* is:

An arbitrary number of dots or line starts

Does "an arbitrary number of line starts" make sense? I guess not and awk might die while trying to reference to the same anchor multiple times. This may very well end up in an infinite loop.

You either want to

  1. Just get rid of the *. This would block metric.gstatic.com and all of its subdomains.
  2. If you want to block anything ending in metric.gstatic.com (so also ABCDEFmetric.gstatic.com), the regex should not have an initial anchor and should look like:
    metric\.gstatic\.com$

What do you really want to achieve with this regex? We can fix it together if you like.

1 Like
28

I went with the more aggressive approach. The only reason the * slipped in there is because I was totally spaced out when entering it and was just thinking I wanted to block *.metric.gstatic.com with * as the wildcard, then just clicked wild card and moved on with my day.

Thanks for the help sorting this one out! Apologies for my poor regex experience.

29

No worries, you know we're always here to help with anything that pops up!

30

I can't duplicate this.

Screenshot_2020-03-02 Pi-hole Admin Console
Screenshot_2020-03-02 Pi-hole Admin Console1530×404 19.1 KB 

root@ubuntu-s-1vcpu-1gb-nyc1-01:~# pihole -q a.metric.gstatic.com
 Match found in regex blacklist
   (\.|^)*\.metric\.gstatic\.com$
31

As the pihole -q issue seems to have the same cause as written here I will mention the outcome here to save prospective readers almost 100 topic replies.

It came down to different versions of awk: the issue occurred only with mawk but not with gawk. So installing gawk fixed it.

Final solution was to circumvent awk in pihole -q and using shell regex checking.
At the moment one can try it with

pihole checkout core fix/awkInQuery
1 Like
32

Fixed in Malformed wildcard blocking doesn't crash awk. by dschaper · Pull Request #3186 · pi-hole/pi-hole · GitHub

1 Like