Pihole 5.14 in Hyper-V DNS Service is not running

Help
1

Expected Behaviour:

Pihole should run normally.

  • Ubuntu 21.10 VM With 2GB of RAM and 1 CPU through Hyper-V
  • Windows 10 Pro as a host

Actual Behaviour:

DNS service not running

Debug Token:

https://tricorder.pi-hole.net/atUOUkqR/

I ran all the possible commands trying to update, repair and restart DNS service. Nothing worked (well everything worked but the result is the same.

Can you please help me figure out what is wrong here?

Many thanks.

LE:

$ pihole restartdns
[sudo] password for iosif: 
  [✗] Job for pihole-FTL.service failed because a timeout was exceeded.
See "systemctl status pihole-FTL.service" and "journalctl -xeu pihole-FTL.service" for details.

$ systemctl status pihole-FTL.service
× pihole-FTL.service - LSB: pihole-FTL daemon
     Loaded: loaded (/etc/init.d/pihole-FTL; generated)
     Active: failed (Result: timeout) since Mon 2022-02-21 09:49:33 EET; 4min 1>
       Docs: man:systemd-sysv-generator(8)
    Process: 5484 ExecStart=/etc/init.d/pihole-FTL start (code=killed, signal=T>
        CPU: 120ms

Feb 21 09:44:33 DNS-3 systemd[1]: Starting LSB: pihole-FTL daemon...
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: .
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: Stopped
Feb 21 09:44:34 DNS-3 su[5501]: (to pihole) root on none
Feb 21 09:44:34 DNS-3 su[5501]: pam_unix(su:session): session opened for user p>
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: start operation timed out>
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: Failed with result 'timeo>
Feb 21 09:49:33 DNS-3 systemd[1]: Failed to start LSB: pihole-FTL daemon.
lines 1-15/15 (END)

iosif@DNS-3:~$ journalctl -xeu pihole-FTL.service
░░ 
░░ A start job for unit pihole-FTL.service has begun execution.
░░ 
░░ The job identifier is 4435.
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: .
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: Stopped
Feb 21 09:44:34 DNS-3 su[5501]: (to pihole) root on none
Feb 21 09:44:34 DNS-3 su[5501]: pam_unix(su:session): session opened for user p>
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: start operation timed out>
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: Failed with result 'timeo>
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit pihole-FTL.service has entered the 'failed' state with result 'time>
Feb 21 09:49:33 DNS-3 systemd[1]: Failed to start LSB: pihole-FTL daemon.
░░ Subject: A start job for unit pihole-FTL.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit pihole-FTL.service has finished with a failure.
░░ 
░░ The job identifier is 4435 and the job result is failed.
lines 916-938/938 (END)
░░ 
░░ A start job for unit pihole-FTL.service has begun execution.
░░ 
░░ The job identifier is 4435.
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: .
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: Stopped
Feb 21 09:44:34 DNS-3 su[5501]: (to pihole) root on none
Feb 21 09:44:34 DNS-3 su[5501]: pam_unix(su:session): session opened for user pihole by (uid=0)
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: start operation timed out. Terminating.
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit pihole-FTL.service has entered the 'failed' state with result 'timeout'.
Feb 21 09:49:33 DNS-3 systemd[1]: Failed to start LSB: pihole-FTL daemon.
░░ Subject: A start job for unit pihole-FTL.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit pihole-FTL.service has finished with a failure.
░░ 
░░ The job identifier is 4435 and the job result is failed.
~
~
lines 916-938/938 (END)
░░ 
░░ A start job for unit pihole-FTL.service has begun execution.
░░ 
░░ The job identifier is 4435.
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: .
Feb 21 09:44:34 DNS-3 pihole-FTL[5484]: Stopped
Feb 21 09:44:34 DNS-3 su[5501]: (to pihole) root on none
Feb 21 09:44:34 DNS-3 su[5501]: pam_unix(su:session): session opened for user pihole by (uid=0)
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: start operation timed out. Terminating.
Feb 21 09:49:33 DNS-3 systemd[1]: pihole-FTL.service: Failed with result 'timeout'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit pihole-FTL.service has entered the 'failed' state with result 'timeout'.
Feb 21 09:49:33 DNS-3 systemd[1]: Failed to start LSB: pihole-FTL daemon.
░░ Subject: A start job for unit pihole-FTL.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit pihole-FTL.service has finished with a failure.
░░ 
░░ The job identifier is 4435 and the job result is failed.
~
~
lines 916-938/938 (END)

I tried to see if my db is having issues, gravity seems fine:

$ pihole -g -r recover
  [✓] Checking integrity of existing gravity database - no errors found
  [✓] Checking foreign keys of existing gravity database - no errors found
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Using libz compression
  [✓] Creating new gravity databases
  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [✓] The old database remains available.
  [i] Number of gravity domains: 1563444 (1297740 unique domains)
  [i] Number of exact blacklisted domains: 18
  [i] Number of regex blacklist filters: 22
  [i] Number of exact whitelisted domains: 117
  [i] Number of regex whitelist filters: 5
  [✗] /usr/local/bin/pihole: line 163: kill: (8688) - No such process
  [✓] Cleaning up stray matter
  [✗] Job for pihole-FTL.service failed because a timeout was exceeded.
See "systemctl status pihole-FTL.service" and "journalctl -xeu pihole-FTL.service" for details.

  [✓] FTL is listening on port 
     [✓] UDP (IPv4)
     [✓] TCP (IPv4)
     [✓] UDP (IPv6)
     [✓] TCP (IPv6)

  [✓] Pi-hole blocking is enabled

After all these steps and a RAM upgrade from 2GB to 4GB, pihole appears ok again but with 17 millions queries in the last 24 hours and they are all coming from my router. Most likely is something wrong with my configuration? I really need some help on this one.

Some details regarding my configuration:
Router: Asus RT-AC68U

  • WAN DNS Set to: 8.8.8.8 and 1.1.1.1
  • LAN (DHCP) DNS Set to: 192.168.0.24 (PiHole's IP). The router takes care of the DHCP part as I didn't wanted to crash all the network if the VM becomes unstable (what happened today).
  • IPv6 Enabled due to the fact that otherwise I would have a non routable IPv4 address setup by my ISP (RO RDS).

PiHole settings:
DNS tab:

  • Upstream servers: Cloudflare (DNSSEC) for IPv4 1, IPv4 2, IPv6 1 and IPv6 2
  • Allow only local requests Checked
  • Never forward non-FQDN A and AAAA queries Unchecked
  • Never forward reverse lookups for private IP ranges Checked
  • Use DNSSEC Unchecked
  • Use Conditional Forwarding
    Local network in CIDR notation 192.168.0.0/24
    IP address of your DHCP server (router) 192.168.0.1
    Local domain name (optional)

System tab:
|FTL version:|v5.14|
|Process identifier (PID):|1764|
|Time FTL started:|Mon Feb 21 10:52:14 2022 EET|
|User / Group:|pihole / pihole|
|Total CPU utilization:|0.0%|
|Memory utilization:|24.4%|
|Used memory:|938.60 MB|
|DNS cache size:|10000|
|DNS cache insertions:|821|
|DNS cache evictions:|0|

Many thanks.

3

Your debug log shows that Pi-hole's DNS service at least has been running sucessfully at some point in the past, but was nearing memory exhaustion due to an extraordinarily excessive amount of DNS queries:

*** [ DIAGNOSING ]: contents of /var/log
-rw-r--r-- 1 pihole pihole 3,2M feb 21 09:46 /var/log/pihole-FTL.log
   -----tail of pihole-FTL.log------
   [2022-02-21 09:46:58.740 5503M] Resizing "FTL-queries" from 915439616 to (16351232 * 56) == 915668992 (/dev/shm: 916.0MB used, 1.0GB total, FTL uses 915.9MB)
   [2022-02-21 09:46:58.741 5503M] WARNING: RAM shortage (/dev/shm) ahead: 91% is used (/dev/shm: 916.0MB used, 1.0GB total, FTL uses 915.9MB)

*** [ DIAGNOSING ]: contents of /dev/shm
-rw------- 1 pihole pihole 348K feb 21 09:44 /dev/shm/FTL-clients
-rw------- 1 pihole pihole 4,0K feb 21 09:44 /dev/shm/FTL-dns-cache
-rw------- 1 pihole pihole 24K feb 21 09:45 /dev/shm/FTL-domains
-rw------- 1 pihole pihole 874M feb 21 09:46 /dev/shm/FTL-queries

The amount of queries from the last 24 hours (at most) already exhausts all of your available memory.

Such an unsually high amount of DNS requests is commonly caused either by a DNS loop or by exposing your Pi-hole publically.

Your debug log contains indications for both of those possible causes.

For the latter, you seem to redirect HTTP requests to your Pi-hole's to HTTPS for a publically reachable domain.
Note that allowing indiscriminate access to Pi-hole from public networks will turn your Pi-hole into an open resolver, and the Pi-hole team strongly discourages Pi-hole’s usage as such an open resolver - we won't provide support in that case.

If you are running your Pi-hole installation on a server with a public IP, please make sure that its properly firewalled and its DNS port is closed to the public.
The recommended way to access a cloud-based Pi-hole is by means of a secure VPN.

As far as DNS loops are concerned, your debug log shows that you have enabled Conditional Forwarding, which could close a partial DNS loop.

This seems more likely in your case, as your RAM usage for domains and clients is reasonably low, whereas only your queries seem vastly excessive, which may suggest a high volume of same domain DNS requests (as caused by a loop).

We could try to verify this by running the following commands on your Pi-hole machine.
Please share the output of:

echo ">stats >quit" | nc localhost 4711

echo ">top-clients >quit" | nc localhost 4711

echo ">top-domains >quit" | nc localhost 4711

BUT those would require an active pihole-FTL, so you may have to reboot your Pi-hole machine.
Even with a reboot, it may time-out when trying to load your excessive queries for the last 24h hours from the database into memory.
I'd therefore recommend to move the bloated query database out of the way before rebooting:

sudo service pihole-FTL stop

sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db

sudo reboot now
4

On a different machine I have a Docker Compose stack. Among other services, I have Trafik configured as a reverse proxy and another service to authenticate through Google for all my subdomains.
I access PiHole through a subdomain when I am not at home. But that is only in order to check the system, not to use it as a DNS resolver.
I've tried to run the 3 commands but for the moment I do not see any result popping out.
Pihole appears to run properly from what I can see in the Dashboard:

Status

Active
Load: 0.04 0.52 0.56
Memory usage: 20.4 %

5

Please share the output anyway.

6

It finally worked:

iosif@DNS-3:~$ echo ">stats >quit" | nc localhost 4711
domains_being_blocked 1297740
dns_queries_today 17872171
ads_blocked_today 821
ads_percentage_today 0.004594
unique_domains 997
queries_forwarded 16435677
queries_cached 21703
clients_ever_seen 13
unique_clients 13
dns_queries_all_types 17872171
reply_NODATA -1661
reply_NXDOMAIN -11
reply_CNAME -250
reply_IP -287
privacy_level 0
status enabled
iosif@DNS-3:~$ 


iosif@DNS-3:~$ echo ">top-clients >quit" | nc localhost 4711
0 17852838 fe80::aa5e:45ff:fe9b:1f10 
1 12233 192.168.0.21 docker-box
2 1956 192.168.0.63 osmc
3 1618 192.168.0.55 IOSIF-BOX
4 966 192.168.0.182 BU-D-DELL01
5 702 192.168.0.9 Galaxy-A9-2018
6 518 192.168.0.153 
7 495 192.168.0.212 OnePlus-7T-Pro
8 275 192.168.0.24 pi.hole
9 221 127.0.0.1 localhost
iosif@DNS-3:~$ 


iosif@DNS-3:~$ echo ">top-domains >quit" | nc localhost 4711
0 17807596 rt-ac68u-1f10
1 29393 153.0.168.192.in-addr.arpa
2 6910 registry-1.docker.io
3 3579 auth.docker.io
4 3528 daisy.ubuntu.com
5 2446 api.snapcraft.io
6 1953 192-168-0-55.4c41cc7fdcec46beb87831f8566ada7f.plex.direct
7 1105 ghcr.io
8 794 connectivity-check.ubuntu.com
9 521 www.google.com
iosif@DNS-3:~$
7

It seems to be your router is excessively requesting resolution for rt-ac68u-1f10.
If you look at Pi-hole's Query Log, I'd expect you to see a mssive amount of requests for that domain.

Disable Conditional Forwarding and see if those requests would cease.

8

Ok, I will do that. I enabled it in order to see the actual devices instead of the IP addresses. Is there an alternative to conditional forwarding and having the router handling the DHCP still?

Thanks.

9

To answer that, we'd better establish it is indeed a loop between Pi-hole's CF and your router.

Were those requests from your router for rt-ac68u-1f10 discontinued after you disabled CF?

Also, what's the output of:

grep rt-ac68u-1f10 -m 1 -B2 /var/log/pihole.log
10

As of now (post disabling the Conditional forwarding) there is no output:

iosif@DNS-3:~$ grep rt-ac68u-1f10 -m 1 -B2 /var/log/pihole.log
iosif@DNS-3:~$
11

Then the log file may just have been rotated.
Please try

zgrep rt-ac68u-1f10 -m 1 -B2 /var/log/pihole.log*
12

Unfortunately, the behavior is the same:

iosif@DNS-3:~$ zgrep rt-ac68u-1f10 -m 1 -B2 /var/log/pihole.log*
iosif@DNS-3:~$
13

Are there any log files at all?

ls -lah /var/log/pihole.log*
14

Yes, they are:

iosif@DNS-3:~$ ls -lah /var/log/pihole.log*
-rw-r--r-- 1 pihole pihole 746K Feb 21 13:22 /var/log/pihole.log
-rw-r--r-- 1 pihole pihole 8.1M Feb 21 10:32 /var/log/pihole.log.1
-rw-r--r-- 1 pihole pihole  15M Feb 21 00:00 /var/log/pihole.log.2.gz
-rw-r--r-- 1 pihole pihole 204K Feb 20 00:00 /var/log/pihole.log.3.gz
-rw-r--r-- 1 pihole pihole 141K Feb 19 09:42 /var/log/pihole.log.4.gz
-rw-r--r-- 1 pihole pihole  16M Feb 19 00:00 /var/log/pihole.log.5.gz
iosif@DNS-3:~$
15

Could be that I misspelled the name we search for.
It should be the very same as reported by the top-domains call.

Alternatively, you could try to find the matching lines by searching for your router's IPv6 instead (as reported by the top-clients call), but you may then have to look for just the lines involving your router's hostname.

16

Here it comes:

iosif@DNS-3:~$ zgrep fe80::aa5e:45ff:fe9b:1f10  -m 1 -B2 /var/log/pihole.log*
/var/log/pihole.log:Feb 21 10:41:23 dnsmasq[11168]: config 24.0.168.192.in-addr.arpa is <PTR>
/var/log/pihole.log:Feb 21 10:41:23 dnsmasq[11168]: query[PTR] 0.1.f.1.b.9.e.f.f.f.5.4.e.5.a.a.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa from 127.0.0.1
/var/log/pihole.log:Feb 21 10:41:23 dnsmasq[11168]: config fe80::aa5e:45ff:fe9b:1f10 is NXDOMAIN
/var/log/pihole.log.1:Feb 21 09:53:26 dnsmasq[8688]: query[PTR] 153.0.168.192.in-addr.arpa from 127.0.0.1
/var/log/pihole.log.1:Feb 21 09:53:27 dnsmasq[8688]: forwarded 153.0.168.192.in-addr.arpa to 192.168.0.1
/var/log/pihole.log.1:Feb 21 09:53:27 dnsmasq[8688]: query[PTR] 153.0.168.192.in-addr.arpa from fe80::aa5e:45ff:fe9b:1f10
/var/log/pihole.log.2.gz:Feb 20 00:47:57 dnsmasq[9125]: query[AAAA] 192-168-0-55.4c41cc7fdcec46beb87831f8566ada7f.plex.direct from 192.168.0.63
/var/log/pihole.log.2.gz:Feb 20 00:47:57 dnsmasq[9125]: cached 192-168-0-55.4c41cc7fdcec46beb87831f8566ada7f.plex.direct is NODATA-IPv6
/var/log/pihole.log.2.gz:Feb 20 00:48:03 dnsmasq[9125]: query[AAAA] edge-mqtt.facebook.com from fe80::aa5e:45ff:fe9b:1f10
/var/log/pihole.log.3.gz:Feb 19 09:43:21 dnsmasq[9125]: read /etc/pihole/custom.list - 0 addresses
/var/log/pihole.log.3.gz:Feb 19 09:43:21 dnsmasq[9125]: read /etc/pihole/local.list - 0 addresses
/var/log/pihole.log.3.gz:Feb 19 09:43:22 dnsmasq[9125]: query[A] p16-sign-va.tiktokcdn.com from fe80::aa5e:45ff:fe9b:1f10
/var/log/pihole.log.4.gz:Feb 19 00:01:02 dnsmasq[3520]: query[AAAA] auth.docker.io from 192.168.0.21
/var/log/pihole.log.4.gz:Feb 19 00:01:02 dnsmasq[3520]: cached auth.docker.io is NODATA-IPv6
/var/log/pihole.log.4.gz:Feb 19 00:01:03 dnsmasq[3520]: query[AAAA] outlook.office365.com from fe80::aa5e:45ff:fe9b:1f10
/var/log/pihole.log.5.gz:Feb 18 00:29:28 dnsmasq[349928]: forwarded youtubei.googleapis.com to 1.1.1.1
/var/log/pihole.log.5.gz:Feb 18 00:29:28 dnsmasq[349928]: reply youtubei.googleapis.com is 142.250.185.170
/var/log/pihole.log.5.gz:Feb 18 00:29:32 dnsmasq[349928]: query[A] connectivity-check.ubuntu.com from fe80::aa5e:45ff:fe9b:1f10
iosif@DNS-3:~$ 

iosif@DNS-3:~$ grep fe80::aa5e:45ff:fe9b:1f10  -m 1 -B2 /var/log/pihole.log
Feb 21 10:41:23 dnsmasq[11168]: config 24.0.168.192.in-addr.arpa is <PTR>
Feb 21 10:41:23 dnsmasq[11168]: query[PTR] 0.1.f.1.b.9.e.f.f.f.5.4.e.5.a.a.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa from 127.0.0.1
Feb 21 10:41:23 dnsmasq[11168]: config fe80::aa5e:45ff:fe9b:1f10 is NXDOMAIN
iosif@DNS-3:~$
17

That's a start, but you'd need to search differently here, as we'd still need the resolution logs for rt-ac68u-1f10 or whatever that is spelled correctly.
If you are not really familiar with grep and sed, it may be easier to use the correct spelling of the hostname instead.

If you can't establish what the correct name would be, you'd have to search for requests from that IPv6, e.g.:

grep -n -e "query\[A.* fe80::aa5e:45ff:fe9b:1f10" /var/log/pihole.log*

Note the file name and the line number of a matching entry.
You'd then have to narrow down that list to requests for that rt-ac68u-1f10 hostname by looking at the corresponding lines after that line number, e.g. for a hit in line number 1000 in pihole.log.1:

sed -n '1000,1010p;1011q' /var/log/pihole.log.1
18

So this morning I woke up and found the below errors in the PiHole interface:

Disk shortage (/var/log/pihole-FTL.log) ahead: 99% used
/var/log: 33.0GB used, 33.1GB total
Disk shortage (/etc/pihole/pihole-FTL.db) ahead: 99% used
/etc/pihole: 33.0GB used, 33.1GB total

So I used the commands from yesterday to make a backup of the FTL db and then I delete it:

sudo service pihole-FTL stop
sudo mv /etc/pihole/pihole-FTL.db /etc/pihole/pihole-FTL-old.db
sudo reboot now
sudo rm /etc/pihole/pihole-FTL-old.db

Here is a fresh debug token. How is my config now?

Thanks.

LE: The interface looks weird:
In the total queries I see only green although I see that there were some blocked ads (12.8%)

Pihole GUI_1
Pihole GUI_11545×1166 173 KB

Pihole GUI_2
Pihole GUI_21261×1212 94.7 KB

19

Your debug log shows that you seem to have allocated more memory and disk space to your VM and you still have enabled Conditional Forwarding.

Increasing your VM resources will not alleviate your issue (it may only make it occur less often, perhaps).

Your issue is an excessive amount of DNS queries, caused by either a partial DNS loop closed by CF or by a misbehaving client.
Your previous top-domain and top-client API call results make the former more likely.

Keep Conditional Forwarding disabled for now and see if this solves your issue.

Your debug log also shows that your router is distributing its own IP as DNS server via DHCP in addition to Pi-hole:

*** [ DIAGNOSING ]: Discovering active DHCP servers (takes 10 seconds)
   Scanning all your interfaces for DHCP servers
   
   * Received 300 bytes from eth0:192.168.0.1
     Offered IP address: 192.168.0.24
     DHCP options:
      dns-server: 192.168.0.24
      dns-server: 192.168.0.1
      router: 192.168.0.1

This will allow clients to by-pass your Pi-hole.
Pi-hole has to be your only DNS server for your clients.

20

Hey there,
I didn't touched the CF since yesterday. This is what I see in the GUI:

DNS
DNS1257×1262 165 KB

The Asus routers are famous for insisting to advertise their own IP address as a DNS unfortunately.
Also, Ubuntu OS seems to see the HDD as yesterday (34GB), weird that Pihole sees it but Ubuntu doesn't.

21

My mistake - I may have mixed up your debug logs when comparing for differences.

That would exclude a CF induced DNS loop from causing your issue, making client behaviour the next suspect.

What does your router return when queried for a name for its IPv4 and IPv6 address?

nslookup 192.168.0.1 192.168.0.1

nslookup fe80::aa5e:45ff:fe9b:1f10 192.168.0.1

They shouldn't trigger any corresponding PTR requests to Pi-hole, but please also watch Pi-hole's Query Log when running those commands.