Because I'm a masochist, I ran it a second time.
This time I'm getting held up on
*** [ DIAGNOSING ]: Info table
I'm getting the feeling that there may be something running in the background during -d
A necessary thing for a true bug fighter!
I'm pretty sure the debugger is a very simple single-task application but Dan might now more.
@Matt Could you watch htop (or nmon or glances or whatever you prefer) during the debugger run? I expect the majority of CPU power to be consumed by sqlite3 following your description, but it's not yet clear what is the real bottleneck.
Away we go!
New -d, new pause points...
*** [ DIAGNOSING ]: Dashboard and block page
SSH has stopped refreshing at this point. Will update if/when it refreshes.
Update: htop refreshed, pihole -d connection has not changed
Thought: It appears they may be related! It looks like the slow -q performance is what's operating in the background slowing down -d
Since I ran it anyways (~30 min), have another debug token: https://tricorder.pi-hole.net/uv26xfowci
Looks like the awk call in /opt/pihole/query.sh -adlist localhost -bp is eating up resources. That ties in to
2020-02-15 09:47:25: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: file(http://127.0.0.1/admin/scripts/pi-hole/php/queryads.php?domain=localhost&bp): failed to open stream: HTTP request failed! in /var/www/html/pihole/index.php on line 134
2020-02-15 09:47:25: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_filter() expects parameter 1 to be array, string given in /var/www/html/pihole/index.php on line 135
2020-02-15 09:47:25: (mod_fastcgi.c.421) FastCGI-stderr: PHP Warning: array_values() expects parameter 1 to be array, null given in /var/www/html/pihole/index.php on line 135
A hostname of localhost -bp doesn't make any sense to me.
This is the line of code that is being triggered: pi-hole/gravity.sh at e41c4b5bb691cea1f5b950d39518d8c404b5846e · pi-hole/pi-hole · GitHub
Somehow ${domain} is being populated by localhost -bp?
1 Like
Are you running pihole -d from the command line or from the web interface?
Command line
I'm not sure then why lighttpd and php are the parent processes for the calls. It looks like https://github.com/pi-hole/AdminLTE/blob/release/v5.0/scripts/pi-hole/php/queryads.php#L64 is the calling process.
1 Like
Just as a test, can you disable the Conditional Forwarding? I think there may be something looping with having a conditionally forwarded domain be the source for a blocklist/blacklist/whitelist.
Testing now, it's stuck, so I'm thinking it didn't change, forgot to pull up htop in a separate session, so I'll loop back with more details in a minute
I'm not sure why pihole -q is looking up localhost, nor why it is being called via lighttpd and php during pihole -d, I'm sure that's it's own problem that this may have brought to light...
Rather than do a full pihole -d (takes about 30 minutes), I ran time pihole -q google.com (only takes 11 minutes) after disabling conditional forwarding. It has not improved performance, and I'm still having the same issue. Here's a screenshot:
The CPU usage fluctuates during the process, but RAM stays completely throttled the entire time.
That code hasn't changed in 6 months or so, but let's pull it apart.
sudo bash -v -x /opt/pihole/query.sh google.com and be prepared for a whole lot of output.
I'm going in! Wish me luck!
Will DM results, as they are likely to be long and probably contain some extra data on my network setup.
Initial result - it seems to be getting hung up while or just after going through my regex list. Will try removing all regex and see if that fixes it.
Alright, it was definitely my regex list. Posting here in case something obviously is wrong:
(^|\.)com\.private-domain\.tld$
(^|\.)net\.private-domain\.tld$
(\.|^)*\.metric\.gstatic\.com$
^(.+[_.-])?ad[sxv]?[0-9]*[_.-]
^(.+[_.-])?adse?rv(er?|ice)?s?[0-9]*[_.-]
^(.+[_.-])?telemetry[_.-]
^(www[0-9]*\.)?xn--
^adim(age|g)s?[0-9]*[_.-]
^adtrack(er|ing)?[0-9]*[_.-]
^advert(s|is(ing|ements?))?[0-9]*[_.-]
^aff(iliat(es?|ion))?[_.-]
^analytics?[_.-]
^banners?[_.-]
^beacons?[0-9]*[_.-]
^count(ers?)?[0-9]*[_.-]
^mads\.
^pixels?[-.]
^stat(s|istics)?[0-9]*[_.-]
^track(ers?|ing)?[0-9]*[_.-]
^traff(ic)?[-.]
The majority of these came from: https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list except:
(^|\.)com\.private-domain\.tld$
(^|\.)net\.private-domain\.tld$
(\.|^)*\.metric\.gstatic\.com$
were built using the wildcard feature through pi-hole
^traff(ic)?[-.] was found on Reddit
Next step is to add these back in one-at-a-time and see which is the problem child.
1 Like
Found the culprit:
(\.|^)*\.metric\.gstatic\.com$
The * appears to break -q.
This is not a valid regular expression. The interpretation of (\.|^)* is:
An arbitrary number of dots or line starts
Does "an arbitrary number of line starts" make sense? I guess not and awk might die while trying to reference to the same anchor multiple times. This may very well end up in an infinite loop.
You either want to
- Just get rid of the
*. This would blockmetric.gstatic.comand all of its subdomains. - If you want to block anything ending in metric.gstatic.com (so also
ABCDEFmetric.gstatic.com), the regex should not have an initial anchor and should look like:metric\.gstatic\.com$
What do you really want to achieve with this regex? We can fix it together if you like.
1 Like
I went with the more aggressive approach. The only reason the * slipped in there is because I was totally spaced out when entering it and was just thinking I wanted to block *.metric.gstatic.com with * as the wildcard, then just clicked wild card and moved on with my day.
Thanks for the help sorting this one out! Apologies for my poor regex experience.
No worries, you know we're always here to help with anything that pops up!
I can't duplicate this.
root@ubuntu-s-1vcpu-1gb-nyc1-01:~# pihole -q a.metric.gstatic.com
Match found in regex blacklist
(\.|^)*\.metric\.gstatic\.com$
As the pihole -q issue seems to have the same cause as written here I will mention the outcome here to save prospective readers almost 100 topic replies.
It came down to different versions of awk: the issue occurred only with mawk but not with gawk. So installing gawk fixed it.
Final solution was to circumvent awk in pihole -q and using shell regex checking.
At the moment one can try it with
pihole checkout core fix/awkInQuery
1 Like