Pi-Hole blocks all clients as if they still used the "default" group

[Watchdog]

Until recently I've had a fairly standard Pi-Hole setup. I have several adlists, several whitelisted domains for my use cases, only the default group and no manually set clients. I also have a Linksys router which forcibly lists itself as a DNS server to DHCP clients instead of sending my Pi-Hole information, however, most devices are manually set to use the Pi-Hole and this should not be the cause of my issue. My Raspberry Pi is also hosting a Wireguard server, but I have had no issues with it and similarly do not believe it be the cause of the issue.

Recently, I've tried to expand my usage of Pi-Hole by introducing group and client management to accommodate those on my network who want to watch ads for various rewards in their games. However, no matter what I do, all the clients, seemingly, still are blocked according to the default group.

Group Config

Client Config

Domain Config

For illustration, let us focus on the blocked domains apple.com, amazon.com, and cloudflare.com. (Don't worry these domains are blocked simply because I needed a domain to test and thought of them first.) Administration and testing is done on the same computer, with an android phone corroborating the same results. nslookup and direct navigation to the websites was used to generate this data.

Theoretically, the following should be true:

• Apple.com should only be blocked to clients in the default group
• Amazon.com should only be blocked to clients in the Test group
• Cloudflare.com should be blocked to clients in both the default and Test group.

Cold, Hard, Reality:

• Apple.com is blocked [Should not be]
• Amazon.com is allowed [Should not be]
• Cloudflare.com is blocked [But for the wrong reasons]

If I swap the groups on apple and amazon, the behavior also swaps, so it seems like the default group is still applying to my devices, even through I have clearly set them to only be apart of the Test group. I have also triple checked the MAC and IP addresses of the clients devices to ensure that devices I am messing with are the devices I am testing and both addresses match in both of my devices.

My logic is clearly impeccable and free from errors... in my eyes at least. What do y'all think?

[rdwebdesign]

Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:

pihole -d

or do it through the Web interface:

Tools > Generate Debug Log

[chrislph]

@Watchdog

• Apple.com is blocked [Should not be]
• Amazon.com is allowed [Should not be]

Using this example... from the client in the Test group try to access these two domains and, once again, observe the 'wrong' behaviour you describe here.

Then inspect the Query Log and see if you can locate these two entries. Are they present? If so, does that confirm that apple.com is blocked (showing in red) and amazon.com is allowed (showing in green)? For the blocked apple entry it would also say "Blocked (exact blacklist)" and you could click on that text to take you to the Domain entry that caused the block, in order to verify.

[Watchdog]

@rdwebdesign

Please upload a debug log...

https://tricorder.pi-hole.net/tXzyDxpt/

@chrislph

Then inspect the Query Log and see if you can locate these two entries. Are they present?

Yes, everything seems to be working the way I have described.
Amazon's query

Apple's query

Cloudflare's query.

Apple and Cloudflare are blocked with exact blocklist while Amazon is allowed through.

[chrislph]

@Watchdog

everything seems to be working the way I have described.

It is worth checking the MAC addresses using Pi-hole's own Tools > Network display. I appreciate you say you have triple-checked the MAC addresses, but it may be that the MAC addresses presented are not the ones you are seeing when you have used your method of checking. Or they may have changed since you checked.

For example iPhones have a MAC randomisation feature which gives a different MAC for each network. This is different to the MAC address you will see when you inspect the settings. This feature is on by default on recent iOS and can be disabled completely or per-network. It's designed to give some privacy when using guest networks.

Another example, I'm running some VMs in VMware Fusion and they present different MAC addresses all the time. I have 16 different MAC addresses showing against a single IP address in my Pi-hole's Network view.

Pi-hole's own network view will show you what it can see. You may need to remove the clients from the Test group and locate them again using the dropdown on that screen, and confirm they match the info from the Network page.

Of course it may be that none of this applies and that something else is going on, but this seems a likely explanation since blocking is behaving as expected if the devices were in the Default group, as your switcharound shows, and that in turn would be explained by you adding one MAC to the Test group and a different MAC being used for the connection, meaning the devices are all obtaining the Default group rules.

[rdwebdesign]

This is expected.

Your screenshot shows a device called "Connor-New-PC", but this device is not on the clients list.

Any device without explicit group will use the Default Group.

The exact domain amazon.com is used only by Test Group (id=1).

[Watchdog]

@chrislph

It is worth checking the MAC addresses using Pi-hole's own Tools > Network display.

I've found the Tools > Network display to be limited, as can be expected. Since I don't use the the Pi-Hole as a dhcp server, nor do I employ Conditional Forwarding, the Pi-Hole cannot accurately resolve the MAC to IP to the hostnames. That why my client list is MAC based since I know those will not be changing.

As for the MAC Spoofing you have described, I know it well. Pretty much every modern Operating System, including my Windows 10 and Android test devices, do it by default. That is why I have disabled them on my network and only use the original, factory-baked addresses. The addresses ending 6E:D8 and 59:6D are from my computer and phone respectively reported by the settings menus in said devices. I have also pinged both devices and looked at the ARP Tables several times over the last few days and the returned mac addresses have not changed. It was a good guess, but I do not believe this is it.

@rdwebdesign

This is expected.

Your screenshot shows a device called "Connor-New-PC", but this device is not on the clients list.

Unfortunately this is a case of the network table being outdated and limited for reasons I have described above. "Connor-New-PC" was the hostname a long while ago, however the device has since been repurposed and the hostname was changed. The MAC and IP addresses associated with "Connor-New-PC" are the same ones associated with my test device "Main-PC".

Nevertheless, my phone, the Samsung above, also displays the same behaviour and it is also in the test group (ID=1)

[Bucking_Horn]

Could you rerun those lookups and, instead of the screenshots showing names, could you share what your /var/log/pihole/pihole.log shows for those requests?

Also, what's the result of running sudo ip neighbour on your Pi-hole host at the time of those lookups?

[Watchdog]

@Bucking_Horn

Also, what's the result of running sudo ip neighbour on your Pi-hole host at the time of those lookups?

Nothing immediately sticks out to me. What about you?

@Bucking_Horn

Could you rerun those lookups and, instead of the screenshots showing names, could you share what your /var/log/pihole/pihole.log shows for those requests?

Okay, I think I got what you were looking for. I used sudo less +G pihole.log
apple.com

cloudflare.com

I was not able to find a result for amazon.com

The test device was 192.168.1.5

Interestingly, the timestamps indicate logging is happening at 00:00, June 1st (UTC+1). The system time is set to 16:00, May 31st (UTC-7). Is this expected behaviour?

I am also beginning to think something may be seriously borked. Both of my testing devices have since turned into "unknown" on my network overview list and their queries are being logged incorrectly under my gateway despite hitting the server directly. Day by day, my network overview list becomes more corrupted with misguided, or just plain false data.

I may have to run a backup then nuke the pi-hole installation. Once I reinstall it I might have to look into just running a dhcp server off of the pi-hole or else attempt conditional forwarding just to keep the client records straight. After that I should be able to restore only the white/blacklists, adlists, Groups, and local DNS records without reintroducing corrupted data.

Thoughts?

[Bucking_Horn]

For future posts, could you share log output as text, please?
That would make it easier to search and reuse for others, while allowing you to redact sensitive details more easily before sharing your output.

@Watchdog

Nothing immediately sticks out to me. What about you?

ip neighbour results are intended to allow us matching the IP from the log to a MAC and determining if other IPs would be associated to that same MAC at the time of nslookup requests.

While the IP is not matching your log details, the latter is the case here, as the c2:<redacted>:db MAC is showing up for two IPs, 192.168.1.5 as well as 192.168.1.6.

Furthermore, that specific MAC address is a locally administered one (as its U/L bit is set), suggesting either MAC address randomisation or some kind of virtualisation software.
(The same would apply to your .245's MAC.)

This somehow contradicts your earlier:

@Watchdog

That is why I have disabled them on my network and only use the original, factory-baked addresses

What kind of device is that c2:<redacted>:db?
An SBC, or maybe some network equipment (e.g. an access point or bridge router)?

As for the logs:

@Watchdog

Interestingly, the timestamps indicate logging is happening at 00:00, June 1st (UTC+1). The system time is set to 16:00, May 31st (UTC-7).

Did you rerun the nslookups as requested, or did you just search through pihole.log?
For a recent lookup, timestamps from the log should match your current time.

@Watchdog

The test device was 192.168.1.5.

@Watchdog

however, most devices are manually set to use the Pi-Hole and this should not be the cause of my issue.

That doesn't seem to have been the case here.
Your log shows DNS requests are orginating from your router's 192.168.1.1, and that would be matched against Pi-hole's default group (which would explain your observation).

That would also suggest that your .5 client either is not manually configured to use Pi-hole for DNS, or the log excerpts are not from the actual time when lookups were issued.

You should verify your time and time zone information on your RPi are correct.
Then rerun those nslookups, monitoring Pi-hole's log via pihole -t or via Tools|Tail pihole.log (or search your logs later, e.g. by sudo grep amazon.com /var/log/pihole/pihole.log).

[Watchdog]

@Bucking_Horn

c2:<redacted>:db

I have no idea what is happening with this address. At times it is associated with up to five IPs, and I cannot find what device is actually using it. It seems to like my .5 and .6 clients, but can be easily associated with more. The MAC addresses both my phone and laptop say they are using are the factory addresses, which are globally administered addresses, 2C and 44, respectively.

@Bucking_Horn

(as its U/L bit is set)

Thank you for that by the way, I did not know of that aspect of MAC addressing.

@Bucking_Horn

Did you rerun the nslookups as requested, or did you just search through pihole.log?
For a recent lookup, timestamps from the log should match your current time.

I am not sure quite what you mean here, but I can do both. I just reran nslookups for the test domains and my system time is showing 16:36 on June 5th. I am in the UTC-7 timezone. The timestamps on the file show 00:36 on June 6th, which is the UTC+1 timezone. When I go to the query logs, they show the correct time of 16:36.

@Bucking_Horn

That doesn't seem to have been the case here.
Your log shows DNS requests are orginating from your router's 192.168.1.1, and that would be matched against Pi-hole's default group (which would explain your observation).

Yes, this is I believe, the root of the issue. Despite being manually configured to use the Pi-Hole (I've set static IP, default gateway, DNS Server addresses), and despite the Pi-Hole logging the queries correctly in the query log, it still somehow believes the queries are coming from the router. This, combined with the ever decreasing accuracy of the network list, leads me to believe that, at least in my situation, nuking the Pi-Hole and clearing all its saved files is the only solution. Then, switch DHCP over from my router to the Pi-Hole to keep it from corrupting itself again.

[Bucking_Horn]

@Watchdog

@Bucking_Horn

Did you rerun the nslookups as requested, or did you just search through pihole.log?

I am not sure quite what you mean here

If you had produced the log lines from a search, then they would not necessarily been for the most current requests, but could have shown older ones.

@Watchdog

(...) my system time is showing 16:36 on June 5th. I am in the UTC-7 timezone. The timestamps on the file show 00:36 on June 6th, which is the UTC+1 timezone.

Does 'my system time' refer to your RPi running Pi-hole?
Run from your Pi-hole machine, what's the output of:

timedatectl status

@Watchdog

@Bucking_Horn

Your log shows DNS requests are orginating from your router's 192.168.1.1, and that would be matched against Pi-hole's default group (which would explain your observation).

Yes, this is I believe, the root of the issue. Despite being manually configured to use the Pi-Hole (I've set static IP, default gateway, DNS Server addresses)

@Watchdog

At times it is associated with up to five IPs, and I cannot find what device is actually using it.

Would all of those be connected via wifi?

Combining those observations, it would seem that your router (or another device in your network, e.g. a dedicated firewall machine or an additional router or access point) intercepts and diverts DNS requests to your router's own resolver at 192.168.1.1, which then would forward DNS requests to its Pi-hole upstream.

In that case, re-installing Pi-hole would do nothing to improve your situtation.

You should try to identify the device at that c2:<redacted>:db, and also consult your router's documentation on its firewall and DNS settings.

[jtmusson]

Just one thought here - are you using any kind of WiFi repeaters in your network?

I used these previously and found that on comparing the Pi-hole network list to the device-reported MAC address, that the WiFi repeater would be transforming the MAC address of each device.

Coupled with roaming between access points, this could mean one device reporting multiple MAC addresses on the Pi-hole.

Apologies if this is an unhelpful diversion, offered in case relevant to tracking down why devices are not appearing as expected to Pi-hole.

[Bucking_Horn]

That is an an attentive observation rather than a diversion.

Any layer-3 switching network equipment can be expected to aggregate traffic by its own MAC. It is the reason why I have insistently been asking about such additional network equipment.