Pihole is forwarding local addresses to "IP".origin.asn.cymru.com many thousand times to upstream DNS resolvers

Hi there,

my Pi-Hole (running as docker container on OpenSUSE 15.2 with no patches outstanding) is behaving very strange since yesterday (2021-05-07 15:30 and later). Normally I have about 60.000 queries per day - but have a look by yourself:

Top-permitted-domains are all my "local-IP-adresses plus origin.asn.cymru.com" that my devices use.
Top-client is my Pi-Hole's docker-IP.

tshark -i br-5dcacaf2848e -f "port 53" now shows a normal behaviour after reverting to v5.7 and rebooting my OpenSUSE:

  Pi-hole version is v5.2.4 (Latest: v5.3.1)
  AdminLTE version is v5.4 (Latest: v5.5)
  FTL version is v5.7 (Latest: v5.8.1)

Recreating the container with latest version didn't "fix" my problem :frowning_face:
I didn't patch or update anything right before this started :thinking:
Pi-Hole was started after reboot on May 2nd and everything was/seems fine until today when Pi-Hole stopped working:

Mai 02 08:11:53 merz-nimbus docker[3884]:  ::: Docker start setup complete
Mai 02 08:11:53 merz-nimbus docker[3884]:   Pi-hole version is v5.3.1 (Latest: v5.3.1)
Mai 02 08:11:53 merz-nimbus docker[3884]:   AdminLTE version is v5.5 (Latest: v5.5)
Mai 02 08:11:53 merz-nimbus docker[3884]:   FTL version is v5.8.1 (Latest: v5.8.1)
Mai 02 08:11:53 merz-nimbus docker[3884]: [cont-init.d] 20-start.sh: exited 0.
Mai 02 08:11:53 merz-nimbus docker[3884]: [cont-init.d] done.
Mai 02 08:11:53 merz-nimbus docker[3884]: [services.d] starting services
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting crond
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting lighttpd
Mai 02 08:11:53 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 02 08:11:53 merz-nimbus docker[3884]: [services.d] done.
Mai 08 07:36:05 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:36:05 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:36:05 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 08 07:51:22 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:51:22 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:51:22 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
08 07:52:34 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:52:34 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:52:34 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root
Mai 08 07:52:49 merz-nimbus docker[3884]: Stopping pihole-FTL
Mai 08 07:52:49 merz-nimbus docker[3884]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 07:52:49 merz-nimbus docker[3884]: Starting pihole-FTL (no-daemon) as root

…
(loop forver with kill/starting/stopping)
…

Questions:

  1. Have you every seens this before?
  2. What is going on there with (my) latest Pi-Hole?

Thanks for any advice and help!
Thomas

P.S. While write this it crashed again - even with older version - so it doesn't seem to be related to Pi-Hole-Version…

Mai 08 09:30:29 merz-nimbus docker[19490]: Stopping pihole-FTL
Mai 08 09:30:29 merz-nimbus docker[19490]: kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
Mai 08 09:30:29 merz-nimbus docker[19490]: Starting pihole-FTL (no-daemon) as root
Mai 08 09:30:46 merz-nimbus docker[19490]: Stopping pihole-FTL
…

Pi-hole isn't forwarding anything to that domain.

You are observing a client -presumably 172.21.0.1- massively requesting resolution of *.origin.asn.cymru.com domains, which look quite similar to reverse lookups.
And indeed, Team Cymru seems to be offering reverse lookup services.

The queries issued are not limited to private IPs, though, there are public IPs in there as well.

Pi-hole wouldn't use those services, nor would it create those requests.

You'd have to take a closer look at the client issuing those requests.

For me it looks like requests are coming from Pi-Hole's IP.
192.168.42.241 is my OpenSUSE host. This IP is to be used as DNS resolver via DHCP-settings:

✔ ~/dev/docker-pi-hole/etc-pihole [master|…5] 
10:13 $ sudo tshark  -f "port 53"|ack cymru
  480   481 51.000818965    127.0.0.1 → 127.0.0.1    DNS 76 Standard query 0xd4e6 A team-cymru.com
  482 51.001032208   172.21.0.1 → 172.21.0.2   DNS 76 Standard query 0xd4e6 A team-cymru.com
  483 51.001051984   172.21.0.1 → 172.21.0.2   DNS 76 Standard query 0xd4e6 A team-cymru.com
  484 51.004586204   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
  485 51.004586204   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
  486 51.004617818 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xc105 A team-cymru.com OPT
  487 51.129691909 84.200.69.80 → 192.168.42.241 DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
  488 51.129720742 84.200.69.80 → 172.21.0.2   DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
  489 51.129727982 84.200.69.80 → 172.21.0.2   DNS 231 Standard query response 0xc105 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com OPT
  490 51.129980541   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
  491 51.129980541   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
  492 51.130018094 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xb82b DS team-cymru.com OPT
  493 51.167475900 84.200.69.80 → 192.168.42.241 DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
528   494 51.167494051 84.200.69.80 → 172.21.0.2   DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
  495 51.167499259 84.200.69.80 → 172.21.0.2   DNS 904 Standard query response 0xb82b DS team-cymru.com NSEC3 RRSIG SOA a.gtld-servers.net RRSIG NSEC3 RRSIG OPT
  496 51.167832570   172.21.0.2 → 172.21.0.1   DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
  497 51.167832570   172.21.0.2 → 172.21.0.1   DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
  498 51.167961676    127.0.0.1 → 127.0.0.1    DNS 220 Standard query response 0xd4e6 A team-cymru.com A 192.0.78.130 A 192.0.78.204 NS ns1.cymru.com NS udns1.ultradns.net NS ns3.cymru.com NS udns2.ultradns.net NS ns2.cymru.com
  499 51.168025271    127.0.0.1 → 127.0.0.1    DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
  500 51.168122133   172.21.0.1 → 172.21.0.2   DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
  501 51.168125289   172.21.0.1 → 172.21.0.2   DNS 76 Standard query 0x7cf4 AAAA team-cymru.com
  502 51.168408460   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
  503 51.168408460   172.21.0.2 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
  504 51.168424386 192.168.42.241 → 84.200.69.80 DNS 87 Standard query 0xac9d AAAA team-cymru.com OPT
  505 51.283451146 84.200.69.80 → 192.168.42.241 DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
  506 51.283479205 84.200.69.80 → 172.21.0.2   DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
  507 51.283486261 84.200.69.80 → 172.21.0.2   DNS 137 Standard query response 0xac9d AAAA team-cymru.com SOA ns1.cymru.com OPT
  508 51.283707764   172.21.0.2 → 172.21.0.1   DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
  509 51.283707764   172.21.0.2 → 172.21.0.1   DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
  510 51.283858130    127.0.0.1 → 127.0.0.1    DNS 126 Standard query response 0x7cf4 AAAA team-cymru.com SOA ns1.cymru.com
…
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5] 
10:17 $ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 70:8b:cd:4d:e8:a4 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 7c:b0:c2:67:02:50 brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 70:8b:cd:4d:e8:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.42.241/24 brd 192.168.42.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 192.168.42.5/24 brd 192.168.42.255 scope global secondary eth1:dns
       valid_lft forever preferred_lft forever
    inet6 2002:c0a8:d:1:d1e4:9dfc:3caa:6367/64 scope global temporary dynamic 
       valid_lft 86179sec preferred_lft 14179sec
    inet6 2002:c0a8:d:1:728b:cdff:fe4d:e8a5/64 scope global dynamic mngtmpaddr 
       valid_lft 86179sec preferred_lft 14179sec
    inet6 fe80::728b:cdff:fe4d:e8a5/64 scope link 
       valid_lft forever preferred_lft forever
5: br-5dcacaf2848e: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:35:04:a7:a7 brd ff:ff:ff:ff:ff:ff
    inet 172.21.0.1/16 brd 172.21.255.255 scope global br-5dcacaf2848e
       valid_lft forever preferred_lft forever
    inet6 fe80::42:35ff:fe04:a7a7/64 scope link 
       valid_lft forever preferred_lft forever
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:f5:99:66:df brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f5ff:fe99:66df/64 scope link 
       valid_lft forever preferred_lft forever

I tried now to (re)move pihole-FTL.db and restart Pi-Hole - working fine now (means: I was immediately able to startup with latest version)… hopefully "forever"…

✔ ~/dev/docker-pi-hole/etc-pihole [master|…5] 
10:11 $ ll pihole-FTL.db*
.rw-r--r-- root root  52 KB 15 seconds ago  pihole-FTL.db
.rw-r--r-- root root 1.0 GB 2 minutes ago   pihole-FTL.db.OLD
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5] 
10:11 $ file pihole-FTL.db*
pihole-FTL.db:     SQLite 3.x database, last written using SQLite version 3035004
pihole-FTL.db.OLD: SQLite 3.x database, last written using SQLite version 3035004
✔ ~/dev/docker-pi-hole/etc-pihole [master|…5] 

:question: Are there any "1-GB-limitations" for the FTL-database that I have overlooked?

BTW: forward reverse lookups for private IP ranges is and was turned off before…

Since you are on Docker, this seems unlikely.
A .1 address is commonly associated with a Docker default gateway, which would mean that any client from outside that specific internal Docker subnet may have sent that request (or the Docker daemon itself, but that seems very unlikely as well).
Your output would suggest that 172.21.0.1 is associated with the Docker internal bridge network you are using. Your Pi-hole may well reside inside that subnet, but it most certainly has a different IP (probably the .2).

From your tshark log, it could be that the source request orginated from your Docker host machine, but that's not entirely sure, as that still could be a redirected request.

That setting has no bearing on those requests.

Note that while they may look similar to reverse lookups and may even serve that same purpose, they are NOT standard reverse lookups as defined by RFC 3425 and RFC 3596. Standard reverse lookups always must end in in-addr.arpa or ip6.arpa for IPv4 and IPv6 respectively.

If you cannot identify the device (or devices) behind 172.21.0.1 straight away, have a closer look at the associated queries. Sometimes, the domains they request may hint at a specific device (e.g. an Amazon Echo could show a preference for Amazon domains).

EDIT:
A common cause for huge volume queries would be a DNS loop.
Or, if you were to host your Pi-hole in a cloud setup, it could be you unintentionally run an open resolver that has been publically discovered only recently.

I think that my FTL-DB got corrupted in some way :thinking: I have renamed it to check it later in another container or on another linux host… I'll keep you informed if I find the root-cause for this or have anything new that might be of interest.

There might be the possibility that my router (which queries my Pi-Hole for my "DMZ" network clients) "bombed" my Pi-Hole with these many reverse-lookups… I'm not sure at all, but I think that this strange behaviour might just be a consequence of my start-stop-looping Pi-Hole due to some problems with my FTL-DB…

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

Today I found the reason on https://github.com/zonemaster/zonemaster-engine/issues/272#issuecomment-269903731.
d3xt3r01 wrote:

I just noticed mtr -z google.com uses cymru in the background to get the AS numbers.

I can confirm: when using mtr with -z parameter for a "longer" time cymru is used.

See also on GitHub for OpenSUSE and Ubuntu Package.

So this post is FYI only :slight_smile:

2 Likes

Thank you for sharing that with us, :slight_smile:
Especially since it confirms that Pi-hole was merely bringing those requests to your attention, rather than causing them.

EDIT: Could you elaborate which piece of software was responsible for those requests?

I've moved your post to the original topic.

1 Like

From the man-page of "mtr"…

CONTACT INFORMATION
       For the latest version, see the mtr web page at ⟨http://www.bitwizard.nl/mtr/⟩
       For patches, bug reports, or feature requests, please open an issue on GitHub at: ⟨https://github.com/traviscross/mtr⟩.

…I got the link to GitHub and did search for "cymru": Repository search results · GitHub - so mtr software in combination with "-z" parameter was responsible for those (TXT-)requests. :grin: