DNS Service is stopping suddenly

Help Community Help
1

Hello All,

i'm facing an issue on my organization becuase i'm using pihole for external DNS filtering and internal DNS is already handled by AD. so the main issue is randomly the pihole FTL service is stopped where it causes issues for our services. below are the output of some log files and i really need to have a permanent fix for this.

pihole_svc@dxb-pihol-01:~$ journalctl -u pihole-FTL
May 24 12:19:41 dxb-pihol-01 systemd[1]: Stopping LSB: pihole-FTL daemon...
May 24 12:19:41 dxb-pihol-01 pihole-FTL[51731]: Not running
May 24 12:19:41 dxb-pihol-01 systemd[1]: Stopped LSB: pihole-FTL daemon.
May 24 12:19:41 dxb-pihol-01 systemd[1]: Starting LSB: pihole-FTL daemon...
May 24 12:19:41 dxb-pihol-01 pihole-FTL[51749]: Not running
May 24 12:19:42 dxb-pihol-01 su[51798]: Successful su for pihole by root
May 24 12:19:42 dxb-pihol-01 su[51798]: + ??? root:pihole
May 24 12:19:42 dxb-pihol-01 su[51798]: pam_unix(su:session): session opened for user pihole by (uid=0)
May 24 12:22:21 dxb-pihol-01 pihole-FTL[51749]: FTL started!
May 24 12:22:21 dxb-pihol-01 systemd[1]: Started LSB: pihole-FTL daemon.

pihole_svc@dxb-pihol-01:~$ echo ">stats >quit" | nc localhost 4711
domains_being_blocked 26877
dns_queries_today 1724330
ads_blocked_today 58887
ads_percentage_today 3.415066
unique_domains 196900
queries_forwarded 1403460
queries_cached 261983
clients_ever_seen 4
unique_clients 4
dns_queries_all_types 1724330
reply_NODATA 459
reply_NXDOMAIN 1582
reply_CNAME 3781
reply_IP 7009
privacy_level 0
status enabled


[2022-05-24 12:19:45.225 51813] Resizing "/FTL-strings" from 733184 to 737280
[2022-05-24 12:19:45.242 51813] Resizing "/FTL-domains" from 589824 to 688128
[2022-05-24 12:19:45.255 51813] Resizing "/FTL-strings" from 737280 to 741376
[2022-05-24 12:19:45.274 51813] Resizing "/FTL-queries" from 5275648 to 5505024
[2022-05-24 12:19:45.292 51813] Resizing "/FTL-strings" from 741376 to 745472
[2022-05-24 12:19:45.332 51813] Resizing "/FTL-strings" from 745472 to 749568
[2022-05-24 12:19:45.368 51813] Resizing "/FTL-strings" from 749568 to 753664
[2022-05-24 12:19:45.402 51813] Resizing "/FTL-strings" from 753664 to 757760
[2022-05-24 12:19:45.446 51813] Resizing "/FTL-queries" from 5505024 to 5734400
[2022-05-24 12:19:45.454 51813] Resizing "/FTL-strings" from 757760 to 761856
[2022-05-24 12:19:45.492 51813] Resizing "/FTL-strings" from 761856 to 765952
[2022-05-24 12:19:45.530 51813] Resizing "/FTL-strings" from 765952 to 770048
[2022-05-24 12:19:45.568 51813] Resizing "/FTL-strings" from 770048 to 774144
[2022-05-24 12:19:45.605 51813] Resizing "/FTL-queries" from 5734400 to 5963776
[2022-05-24 12:19:45.607 51813] Resizing "/FTL-strings" from 774144 to 778240
[2022-05-24 12:19:45.636 51813] Resizing "/FTL-strings" from 778240 to 782336
[2022-05-24 12:19:45.682 51813] Resizing "/FTL-strings" from 782336 to 786432
[2022-05-24 12:19:45.715 51813] Resizing "/FTL-strings" from 786432 to 790528
[2022-05-24 12:19:45.749 51813] Resizing "/FTL-strings" from 790528 to 794624
[2022-05-24 12:19:45.762 51813] Resizing "/FTL-queries" from 5963776 to 6193152
[2022-05-24 12:19:45.788 51813] Resizing "/FTL-strings" from 794624 to 798720
[2022-05-24 12:19:45.835 51813] Resizing "/FTL-strings" from 798720 to 802816
[2022-05-24 12:19:45.871 51813] Resizing "/FTL-strings" from 802816 to 806912
[2022-05-24 12:19:45.901 51813] Resizing "/FTL-strings" from 806912 to 811008
[2022-05-24 12:19:45.938 51813] Resizing "/FTL-strings" from 811008 to 815104
[2022-05-24 12:19:45.939 51813] Resizing "/FTL-queries" from 6193152 to 6422528
[2022-05-24 12:19:45.975 51813] Resizing "/FTL-strings" from 815104 to 819200
[2022-05-24 12:19:46.011 51813] Resizing "/FTL-strings" from 819200 to 823296
[2022-05-24 12:19:46.050 51813] Resizing "/FTL-strings" from 823296 to 827392
[2022-05-24 12:19:46.083 51813] Resizing "/FTL-strings" from 827392 to 831488
[2022-05-24 12:19:46.102 51813] Resizing "/FTL-queries" from 6422528 to 6651904
[2022-05-24 12:19:46.117 51813] Resizing "/FTL-strings" from 831488 to 835584
[2022-05-24 12:19:46.154 51813] Resizing "/FTL-strings" from 835584 to 839680
[2022-05-24 12:19:46.194 51813] Resizing "/FTL-strings" from 839680 to 843776
[2022-05-24 12:19:46.235 51813] Resizing "/FTL-strings" from 843776 to 847872
[2022-05-24 12:19:46.252 51813] Resizing "/FTL-queries" from 6651904 to 6881280
[2022-05-24 12:19:46.273 51813] Resizing "/FTL-strings" from 847872 to 851968
[2022-05-24 12:19:46.307 51813] Resizing "/FTL-strings" from 851968 to 856064
[2022-05-24 12:19:46.348 51813] Resizing "/FTL-strings" from 856064 to 860160
[2022-05-24 12:19:46.379 51813] Resizing "/FTL-strings" from 860160 to 864256
[2022-05-24 12:19:46.412 51813] Resizing "/FTL-queries" from 6881280 to 7110656
[2022-05-24 12:19:46.422 51813] Resizing "/FTL-strings" from 864256 to 868352
[2022-05-24 12:19:46.464 51813] Resizing "/FTL-strings" from 868352 to 872448
[2022-05-24 12:19:46.487 51813] Resizing "/FTL-domains" from 688128 to 786432
[2022-05-24 12:19:46.509 51813] Resizing "/FTL-strings" from 872448 to 876544
[2022-05-24 12:19:46.547 51813] Resizing "/FTL-strings" from 876544 to 880640
[2022-05-24 12:19:46.564 51813] Resizing "/FTL-queries" from 7110656 to 7340032
[2022-05-24 12:19:46.576 51813] Resizing "/FTL-strings" from 880640 to 884736
[2022-05-24 12:19:46.603 51813] Resizing "/FTL-strings" from 884736 to 888832
[2022-05-24 12:19:46.632 51813] Resizing "/FTL-strings" from 888832 to 892928
[2022-05-24 12:19:46.668 51813] Resizing "/FTL-strings" from 892928 to 897024
[2022-05-24 12:19:46.712 51813] Resizing "/FTL-strings" from 897024 to 901120
[2022-05-24 12:19:46.714 51813] Resizing "/FTL-queries" from 7340032 to 7569408
[2022-05-24 12:19:46.748 51813] Resizing "/FTL-strings" from 901120 to 905216
[2022-05-24 12:19:46.789 51813] Resizing "/FTL-strings" from 905216 to 909312
[2022-05-24 12:19:46.826 51813] Resizing "/FTL-strings" from 909312 to 913408
[2022-05-24 12:19:46.867 51813] Resizing "/FTL-strings" from 913408 to 917504
[2022-05-24 12:19:46.903 51813] Resizing "/FTL-queries" from 7569408 to 7798784
[2022-05-24 12:19:46.923 51813] Resizing "/FTL-strings" from 917504 to 921600
[2022-05-24 12:19:46.963 51813] Resizing "/FTL-strings" from 921600 to 925696
[2022-05-24 12:19:47.010 51813] Resizing "/FTL-strings" from 925696 to 929792
[2022-05-24 12:19:47.048 51813] Resizing "/FTL-strings" from 929792 to 933888
[2022-05-24 12:19:47.086 51813] Resizing "/FTL-strings" from 933888 to 937984
[2022-05-24 12:19:47.093 51813] Resizing "/FTL-queries" from 7798784 to 8028160
[2022-05-24 12:19:47.120 51813] Resizing "/FTL-strings" from 937984 to 942080
[2022-05-24 12:19:47.159 51813] Resizing "/FTL-strings" from 942080 to 946176
[2022-05-24 12:19:47.202 51813] Resizing "/FTL-strings" from 946176 to 950272
[2022-05-24 12:19:47.232 51813] Resizing "/FTL-strings" from 950272 to 954368
[2022-05-24 12:19:47.268 51813] Resizing "/FTL-strings" from 954368 to 958464
[2022-05-24 12:19:47.311 51813] Resizing "/FTL-strings" from 958464 to 962560
[2022-05-24 12:19:47.315 51813] Resizing "/FTL-queries" from 8028160 to 8257536


pihole_svc@dxb-pihol-01:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            927M     0  927M   0% /dev
tmpfs           192M  1.2M  191M   1% /run
/dev/sda2        40G   14G   24G  36% /
tmpfs           959M  105M  855M  11% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           959M     0  959M   0% /sys/fs/cgroup
/dev/loop2      112M  112M     0 100% /snap/core/12941
/dev/loop1      111M  111M     0 100% /snap/core/12834
tmpfs           192M     0  192M   0% /run/user/999
tmpfs           192M     0  192M   0% /run/user/1000
2

Please post the token URL from pihole -d. Thanks!

3

Thanks for your reply and so sorry for being late. can i do debug on working hours where all users are using internet or should i plan for a window? i dont want to add extra load on the services.

4

Generating the debug log should not affect your DNS resolution.

5

its https://tricorder.pi-hole.net/SU6zh4x4/

6 
*** [ DIAGNOSING ]: Core version
[i] Core: v4.4 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: master
[i] Commit: v4.4-0-g9e49077

*** [ DIAGNOSING ]: Web version
[i] Web: v4.3.3 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)
[i] Branch: master
[i] Commit: v4.3.3-0-g62f2ffc

*** [ DIAGNOSING ]: FTL version
[âś“] FTL: v4.3.1 (https://discourse.pi-hole.net/t/how-do-i-update-pi-hole/249)

Maybe it is time to upgrade before continue troubleshooting.

P.S. There are a lot of changes between your version and the current one. I recommend you

for that.

7

this is what i thought earlier so i created new pihole servers with 5.6 version but and added it to the forwarders in AD but got the same issue again. FTL service is stopping but less frequent to be honest on the new one.

check this token https://tricorder.pi-hole.net/6ql3YNgm/

8

From your debug log I can see, that some of your clients were rate-limited because they exceeded the number of allowed queries within 60 second.

*** [ DIAGNOSING ]: Pi-hole diagnosis messages
   id    timestamp            type                  message                                                       blob1                 blob2                 blob3                 blob4                 blob5               
   ----  -------------------  --------------------  ------------------------------------------------------------  --------------------  --------------------  --------------------  --------------------  --------------------
   2500  2022-06-13 16:02:04  RATE_LIMIT            10.xx.xx.12                                                   1000                  60                                                                                    
   2511  2022-06-14 10:20:34  RATE_LIMIT            10.xx.xx.11                                                   1000                  60

You seem to have configured your network in a way, that all clients send their queries first to AD which will forward them to Pihole. This also explains the low number of clients seen here:

_

You should lift the rate-limit if you wish to keep the order of DNS server within your network as they are now.

https://docs.pi-hole.net/ftldns/configfile/#rate_limit

1 Like
9

Alternatively, you may want to consider swapping Pi-hole and your AD in your DNS resolution chain, i.e. client -> Pi-hole -> AD -> upstream DNS.

10

Thanks for your reply but what will be the advantage of changing the structure like this?

11

can you please hide the IP addresses ?

12

These are private range IP's, the same as many other users on their private LANs.

13

i know but i dont want them to be on public.

14

As of now, your AD (or multiple ADs?) aggregate DNS traffic from your entire network and forward it to Pi-hole.
In consequence, if Pi-hole is rate limiting your AD, all of your AD's clients are affected equally and at once.

If Pi-hole is first in your DNS resolution chain, Pi-hole's per-client rate limiting would occur less often, and if it kicks in, it would only affect a single excessive client (and domains blocked by Pi-hole would be blocked (measurably, but probably insignificantly) faster, as it would occur one hop earlier).

Note that AD controllers may be able to provide other name resolution services beside DNS, e.g. WINS or NetBIOS name resolution.
I am not familiar enough with AD services, so I wouldn't know whether they would offer sufficiently fine-grained control over which name resolution service should be offered by which host, and I am also not at all familiar with your network's potential dependencies on those alternative name resolution services, so I wouldn't know whether or how they matter when you would opt to switch DNS to be Pi-hole first.

15

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.