Exactly--you nailed it. Great summary! 
Great! So I have one remaining question and then we should leave the stage for the experts:
I do understand add-subnet=32,128, but why add-mac (I see you marked it as potential)?
Great question, and to be honest, I'm not sure if it would be needed. I see NextDNS client references it here: nextdns/query.go at 89739d863f346f8935c5b5b9b96c148a5010e948 · nextdns/nextdns · GitHub. I am guessing they use it as a fallback to do a "last-ditch" ARP for the IP in the case where the subnet did not come over in the ECS record.
** Update **
Thinking through this a bit more, it would stand to reason that add-mac would also come into play if the ECS is > /32 for IPv4 or > /128 for IPv6. If either of those cases is true, what was received in the ECS would be a true subnet as opposed to a single host IP (e.g. FTLDNS gets a 192.168.0.0/24 from ECS instead of 192.168.0.57/32). That would not be helpful for narrowing the query down to a single IP/hostname, so it seems ARPing for the provided MAC would be the next available option for obtaining a single IP.
but would certainly also introduce more work = delay
Agreed, but it might be a trade-off one is willing to make in order to get a more clear picture of the device making the query. I'm of the mindset all of this, even if implemented, would be more of an advanced/opt-in type of configuration. For those that understand it and opt for using it, they could make that judgement call for the added milliseconds of an additional ARP.
My expectation would be that FTLDNS would also strip ECS from the query before forwarding upstream for security reasons. But even that could be an opt-in/opt-out feature.
I must say I like this idea and see a lot of potential value in it (given routers can be made to send ECS data).
Over the past two hours, I went back and forth between RFC 6891 and RFC 7871 and finished a first implementation of the Extension Mechanisms for DNS ("EDNS(0)"). FTL should now be able to extract the proper address + prefix-length from the incoming queries.
I'm not yet sold on the MAC feature as there doesn't seem to be a lot of publicly available documentation. You can find details spread across mailing lists mentioning unpleasant things like
its value is implementation specific
@_FailSafe It would be great if you could do some initial testing, whether this works as expected in your environment.
Please run
pihole checkout ftl new/edns0
and continue using Pi-hole as usual. Check /var/log/pihole-FTL.log for lines starting in EDNS0.
Note: This is only the very first step. There is much work left to be done because FTL is already interpreting the DNS packet while it is received (for performance reasons).
This has the consequence that everything (like detecting if a given domain should be blocked for a given client, etc.) is already done and set before the additional records of the query (containing the EDNS0 information) are received/analyzed...
3 Likes
I pulled that branch on both of my Pi-holes. Here's a sample of what I'm seeing in the pihole-FTL logs now:
[2020-07-20 15:23:15.738 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:15.738 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:70a9:cedb:910a:c690/128
[2020-07-20 15:23:15.738 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:15.738 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:15.739 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:70a9:cedb:910a:c690/128
[2020-07-20 15:23:15.739 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:15.781 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:15.781 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:70a9:cedb:910a:c690/128
[2020-07-20 15:23:15.781 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.106 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.106 57621M] EDNS0: Identified option CLIENT SUBNET 192.168.99.120/32
[2020-07-20 15:23:16.106 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.357 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.357 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:1cce:b4b0:9702:281c/128
[2020-07-20 15:23:16.357 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.357 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.357 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:1cce:b4b0:9702:281c/128
[2020-07-20 15:23:16.357 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.358 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.358 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:1cce:b4b0:9702:281c/128
[2020-07-20 15:23:16.358 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.500 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.500 57621M] EDNS0: Identified option CLIENT SUBNET 192.168.99.120/32
[2020-07-20 15:23:16.500 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.501 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.501 57621M] EDNS0: Identified option CLIENT SUBNET 192.168.99.120/32
[2020-07-20 15:23:16.501 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.502 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.502 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2309:1110:c38d:5708:7fb6/128
[2020-07-20 15:23:16.502 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:16.502 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:16.502 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2309:1110:c38d:5708:7fb6/128
[2020-07-20 15:23:16.502 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:f41d:94bd:fd03:a09f/128
[2020-07-20 15:23:17.318 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:f41d:94bd:fd03:a09f/128
[2020-07-20 15:23:17.318 57621M] EDNS0: Originally recorded client was 192.168.50.5
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option MAC ADDRESS
[2020-07-20 15:23:17.318 57621M] EDNS0: Identified option CLIENT SUBNET XXXX:YYYY:ZZZZ:2345:f41d:94bd:fd03:a09f/128
[2020-07-20 15:23:17.318 57621M] EDNS0: Originally recorded client was 192.168.50.5
This is working exactly as I would expect, to be honest. Let me know what I can do to further help in bringing this feature into reality. I am very happy you feel it is worth the time and effort. 
2 Likes
Remaining steps done as well. It was a bit painful to rearrange the DNS header analysis stuff, but I think I managed to do it without breaking anything. Please update again and check if it works now. The next steps are
- monitor for strange side-effects (I actually don't really expect any), and
- disable the debugging output you've quoted by default.
This definitively is going to be in the next version of Pi-hole (well, not the one about to be released, but the one after that) if nothing strange comes up in further tests.
1 Like
This is fantastic, really. I did update again and both Pi-holes are working beautifully now. I have my FTLDNS configuration set up to do reverse IPv4 and IPv6 lookups against dnsmasq on my OpenWrt box. Because I also run ip6neigh on my OpenWrt box, I have IPv6 SLAAC reverse lookups working there.
So with the updates you just made for EDNS0 processing, I am getting my ipsets populated on OpenWrt for QoS purposes, client IP/hostname reporting against queries in my Pi-hole VMs, and the piece of mind of having my Pi-holes querying externally via dnscrypt-proxy. I'm a happy camper with a big smile on my face right now. 
View of Pi-hole web console:
(Look at all that samsung tracking getting blocked...
)
I will obviously be using the heck out of this, so I will be sure to report back on any hiccups. Although I have a good feeling this will be a continued success!
2 Likes
Another candidate for the most awesome new feature of the year. Pi-hole and especially their core developers definitely deserve credit and recognition for the awesome work they do almost every day!
COME ON. This (complex!) feature just been requested only yesterday and they implemented it on the same day. And the user requesting this already replied that this is doing exactly what they wanted it to do.
You cannot tell me that this is what you can expect from an average open source project.
Run by volunteers.
You are absolutely top notch guys!
Service, quality and support - everything just rocks in this project.
2 Likes
So I invested some more work into this to "finish" the feature. We can now also parse EDNS(0) cookies and a few other things even if we don't do anything with them (yet).
Concerning MAC addresses, this is really not standardized at all, I implemented it now in the exact same way dnsmasq has it. It is very possible that dnsmasq is the only DNS client that sends this packet. I implemented support for add-mac and add-mac=text. The former is more efficient, so to be preferred. I did not add support for add-mac=base64 as I couldn't be bother to code a base64 decoder. I don't think this is much of a loss.
It would be nice if you could update and do some more testing - are MAC addresses correctly interpreted and shown? Please add
DEBUG_EDNS0=true
to /etc/pihole/pihole-FTL.conf (create if it does not exist) so you get the debug log lines.
1 Like
Confirmed. I spot checked around two dozen MAC/IP combinations from the logs and the interpreted MAC addresses marry up to both IPv4 and IPv6 addresses perfectly. I validated the matches based on the arp table and ip6neigh whois command on my OpenWrt box.
I did not modify my OpenWrt dnsmasq configuration to add-mac=text, but could do so if you would like me to test that specifically.
4 Likes
No need to. Thanks for testing this so thoroughly!
1 Like
Absolutely! I appreciate the time and attention to this. Testing is the least I can do. If you have any other test-cases you would like me to run through, I'll be very happy to do that.
For what it's worth, it is still humming along very nicely. I thought maybe it hit an issue last evening because one of my Pi-holes got stuck in a loop trying to resolve an internal IPv6 PTR record. It ended up taking both of my Pi-holes to their knees. Was taking around 4 seconds per query to resolve any requests from my client machines.
Turned out to be a self-inflicted DNS forwarding loop with my OpenWrt dnsmasq. I forgot to modify my OWRT dnsmasq config to not forward internal IPv6 PTR lookups since it handles those locally via ip6neigh. Anyway, 15 mins of troubleshooting and >120,000 DNS queries later, I limped away from that one with a bruised ego. But, I am very happy it was not related to the changes you made for EDNS0 handling. 
** 48+ Hours Update **
Still running smoothly as can be on both of my Pi-holes. No issues to report.
6 Likes
Zero issues to report on this branch. Both Pi-holes have been running solid. I will not keep updating this thread with uptime reports unnecessarily, but still happy to do any additional testing needed.
@DL6ER Thanks again for the efforts put into this. I think it will be a great feature that will simplify a lot of configurations going forward!
3 Likes
@DL6ER Would it be feasible to have the client subnet included in the query[<record type>] <fqdn> from <requester IP> output in pihole.log (and subsequently via pihole -t)?
Possible: Yes
Feasible: Absolutely no.
The reason for this is that we added an additional field in the Pi-hole related part of the daemon. The dnsmasq.log is, however, written entirely and solely by the embedded dnsmasq core which is not aware of this EDNS trick. It known only about one client for a query and that is (and has to be) the client which made the request (= is expecting the reply). Changing this would disturb just too much in the code and would cause unpleasant merge issues when future versions of dnsmasq are to be embedded/upgraded.
1 Like
Understood and I figured this would be the case. Not a big deal at all—just thought I would ask while on this topic. 
Thanks!
My setup uses dnsdist to forward queries to Pi-hole. Although I'm not doing anything MAC related, I can say that the client subnet recognition is working for me as well.
So you're saying dnsdist attaches EDNS0 Client subnet information as well? Interesting, thanks for sharing this!
2 Likes