my Pi-Hole (running as docker container on OpenSUSE 15.2 with no patches outstanding) is behaving very strange since yesterday (2021-05-07 15:30 and later). Normally I have about 60.000 queries per day - but have a look by yourself:
tshark -i br-5dcacaf2848e -f "port 53" now shows a normal behaviour after reverting to v5.7 and rebooting my OpenSUSE:
Pi-hole version is v5.2.4 (Latest: v5.3.1)
AdminLTE version is v5.4 (Latest: v5.5)
FTL version is v5.7 (Latest: v5.8.1)
Recreating the container with latest version didn't "fix" my problem
I didn't patch or update anything right before this started
Pi-Hole was started after reboot on May 2nd and everything was/seems fine until today when Pi-Hole stopped working:
Mai 02 08:11:53 merz-nimbus docker[3884]: ::: Docker start setup complete
Mai 02 08:11:53 merz-nimbus docker[3884]: Pi-hole version is v5.3.1 (Latest: v5.3.1)
Mai 02 08:11:53 merz-nimbus docker[3884]: AdminLTE version is v5.5 (Latest: v5.5)
Mai 02 08:11:53 merz-nimbus docker[3884]: FTL version is v5.8.1 (Latest: v5.8.1)
Mai 02 08:11:53 merz-nimbus docker[3884]: [cont-init.d] 20-start.sh: exited 0.
Mai 02 08:11:53 merz-nimbus docker[3884]: [cont-init.d] done.
Mai 02 08:11:53 merz-nimbus docker[3884]: [services.d] starting services
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting crond
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting lighttpd
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 02 08:11:53 merz-nimbus docker[3884]: [services.d] done.
Mai 08 07:36:05 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:36:05 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:36:05 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 08 07:51:22 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:51:22 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:51:22 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
08 07:52:34 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:52:34 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:52:34 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 08 07:52:49 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:52:49 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:52:49 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
…
(loop forver with kill/starting/stopping)
…
Questions:
Have you every seens this before?
What is going on there with (my) latest Pi-Hole?
Thanks for any advice and help!
Thomas
P.S. While write this it crashed again - even with older version - so it doesn't seem to be related to Pi-Hole-Version…
Mai 08 09:30:29 merz-nimbus docker[19490]: Stopping pihole-FTL
Mai 08 09:30:29 merz-nimbus docker[19490]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 09:30:29 merz-nimbus docker[19490]: Starting pihole-FTL (no-daemon) as root
Mai 08 09:30:46 merz-nimbus docker[19490]: Stopping pihole-FTL
…
You are observing a client -presumably 172.21.0.1- massively requesting resolution of *.origin.asn.cymru.com domains, which look quite similar to reverse lookups.
And indeed, Team Cymru seems to be offering reverse lookup services.
The queries issued are not limited to private IPs, though, there are public IPs in there as well.
Pi-hole wouldn't use those services, nor would it create those requests.
You'd have to take a closer look at the client issuing those requests.
For me it looks like requests are coming from Pi-Hole's IP. 192.168.42.241 is my OpenSUSE host. This IP is to be used as DNS resolver via DHCP-settings:
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5]
10:13 $ sudo tshark -f "port 53"|ack cymru
480 481 51.000818965 127.0.0.1 → 127.0.0.1 DNS 76 Standard query 0xd4e6 A team-cymru.com
482 51.001032208 172.21.0.1 → 172.21.0.2 DNS 76 Standard query 0xd4e6 A team-cymru.com
483 51.001051984 172.21.0.1 → 172.21.0.2 DNS 76 Standard query 0xd4e6 A team-cymru.com
484 51.004586204 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
485 51.004586204 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
486 51.004617818 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
487 51.129691909 84.200.69.80 → 192.168.42.241 DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
488 51.129720742 84.200.69.80 → 172.21.0.2 DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
489 51.129727982 84.200.69.80 → 172.21.0.2 DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
490 51.129980541 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
491 51.129980541 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
492 51.130018094 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
493 51.167475900 84.200.69.80 → 192.168.42.241 DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
528 494 51.167494051 84.200.69.80 → 172.21.0.2 DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
495 51.167499259 84.200.69.80 → 172.21.0.2 DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
496 51.167832570 172.21.0.2 → 172.21.0.1 DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
497 51.167832570 172.21.0.2 → 172.21.0.1 DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
498 51.167961676 127.0.0.1 → 127.0.0.1 DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
499 51.168025271 127.0.0.1 → 127.0.0.1 DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
500 51.168122133 172.21.0.1 → 172.21.0.2 DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
501 51.168125289 172.21.0.1 → 172.21.0.2 DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
502 51.168408460 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
503 51.168408460 172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
504 51.168424386 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
505 51.283451146 84.200.69.80 → 192.168.42.241 DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
506 51.283479205 84.200.69.80 → 172.21.0.2 DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
507 51.283486261 84.200.69.80 → 172.21.0.2 DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
508 51.283707764 172.21.0.2 → 172.21.0.1 DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
509 51.283707764 172.21.0.2 → 172.21.0.1 DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
510 51.283858130 127.0.0.1 → 127.0.0.1 DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
…
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5]
10:17 $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 70:8b:cd:4d:e8:a4 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 7c:b0:c2:67:02:50 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 70:8b:cd:4d:e8:a5 brd ff:ff:ff:ff:ff:ff
inet 192.168.42.241/24 brd 192.168.42.255 scope global eth1
valid_lft forever preferred_lft forever
inet 192.168.42.5/24 brd 192.168.42.255 scope global secondary eth1:dns
valid_lft forever preferred_lft forever
inet6 2002:c0a8:d:1:d1e4:9dfc:3caa:6367/64 scope global temporary dynamic
valid_lft 86179sec preferred_lft 14179sec
inet6 2002:c0a8:d:1:728b:cdff:fe4d:e8a5/64 scope global dynamic mngtmpaddr
valid_lft 86179sec preferred_lft 14179sec
inet6 fe80::728b:cdff:fe4d:e8a5/64 scope link
valid_lft forever preferred_lft forever
5: br-5dcacaf2848e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:35:04:a7:a7 brd ff:ff:ff:ff:ff:ff
inet 172.21.0.1/16 brd 172.21.255.255 scope global br-5dcacaf2848e
valid_lft forever preferred_lft forever
inet6 fe80::42:35ff:fe04:a7a7/64 scope link
valid_lft forever preferred_lft forever
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:f5:99:66:df brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:f5ff:fe99:66df/64 scope link
valid_lft forever preferred_lft forever
I tried now to (re)move pihole-FTL.db and restart Pi-Hole - working fine now (means: I was immediately able to startup with latest version)… hopefully "forever"…
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5]
10:11 $ ll pihole-FTL.db*
.rw-r--r-- root root 52 KB 15 seconds ago pihole-FTL.db
.rw-r--r-- root root 1.0 GB 2 minutes ago pihole-FTL.db.OLD
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5]
10:11 $ file pihole-FTL.db*
pihole-FTL.db: SQLite 3.x database, last written using SQLite version 3035004
pihole-FTL.db.OLD: SQLite 3.x database, last written using SQLite version 3035004
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5]
Are there any "1-GB-limitations" for the FTL-database that I have overlooked?
Since you are on Docker, this seems unlikely.
A .1 address is commonly associated with a Docker default gateway, which would mean that any client from outside that specific internal Docker subnet may have sent that request (or the Docker daemon itself, but that seems very unlikely as well).
Your output would suggest that 172.21.0.1 is associated with the Docker internal bridge network you are using. Your Pi-hole may well reside inside that subnet, but it most certainly has a different IP (probably the .2).
From your tshark log, it could be that the source request orginated from your Docker host machine, but that's not entirely sure, as that still could be a redirected request.
That setting has no bearing on those requests.
Note that while they may look similar to reverse lookups and may even serve that same purpose, they are NOT standard reverse lookups as defined by RFC 3425 and RFC 3596. Standard reverse lookups always must end in in-addr.arpa or ip6.arpa for IPv4 and IPv6 respectively.
If you cannot identify the device (or devices) behind 172.21.0.1 straight away, have a closer look at the associated queries. Sometimes, the domains they request may hint at a specific device (e.g. an Amazon Echo could show a preference for Amazon domains).
EDIT:
A common cause for huge volume queries would be a DNS loop.
Or, if you were to host your Pi-hole in a cloud setup, it could be you unintentionally run an open resolver that has been publically discovered only recently.
I think that my FTL-DB got corrupted in some way I have renamed it to check it later in another container or on another linux host… I'll keep you informed if I find the root-cause for this or have anything new that might be of interest.
There might be the possibility that my router (which queries my Pi-Hole for my "DMZ" network clients) "bombed" my Pi-Hole with these many reverse-lookups… I'm not sure at all, but I think that this strange behaviour might just be a consequence of my start-stop-looping Pi-Hole due to some problems with my FTL-DB…
Thank you for sharing that with us,
Especially since it confirms that Pi-hole was merely bringing those requests to your attention, rather than causing them.
EDIT: Could you elaborate which piece of software was responsible for those requests?
CONTACT INFORMATION
For the latest version, see the mtr web page at ⟨http://www.bitwizard.nl/mtr/⟩
For patches, bug reports, or feature requests, please open an issue on GitHub at: ⟨https://github.com/traviscross/mtr⟩.
…I got the link to GitHub and did search for "cymru": Repository search results · GitHub - so mtr software in combination with "-z" parameter was responsible for those (TXT-)requests.