Use-application-dns.net found in adlists

new issue, see here.

1 Like

What we have (and had to) consider is how to handle users that want the canary to resolve and enable the feature on their setups? From my recollection of the discussion, this behavior is exactly intended.

You can add this domain to a group if you wanted and have that group configured to use FF's DoH.

So, yes, what is happening is what should be happening.

Side note: The maintainers of the list in question are the same people that tried to sell Pi-hole devices and then rebranded when their scammy behavior came to light. I think they also allow companies to "sponsor" their lists?

1 Like

Apologies, I don't understand what you're saying.

  • pihole development added ' server=/use-application-dns.net/ ' to /etc/dnsmasq.d/01-pihole.conf to ensure a query (from firefox) got the mandatory NXDOMAIN reply, this to ensure firefox would NOT bypass pihole, by using DoH.
  • whatever construction you setup with groups or other gravity methods, while using the default blocking mode (NULL), the reply for use-application-dns.net will always be 0.0.0.0, which would than allow firefox to use DoH, making having a pihole useless (when using firefox).
    Are there really users that have pihole but don't want firefox to use it? If so, they can always add a dnsmasq custom config to ensure firefox is allowed to use DoH (or modify the default entry in /etc/dnsmasq.d/01-pihole.conf. They would have already made configuration changes to allow this, so probably knowledgeable users.

I've already taken precautions, to ensure the entry never makes it into gravity. Just trying to help and protect other users, that may not even be aware of OR notice a blocklist entry allows firefox to use DoH.

By default, with the ability to override our decision with what the user actually wants to happen.

Yes, this is the same as putting users in a group that get no filtering. Users have to have the ability to disable a function if they so desire.

Yes, and that is their choice.

Going a step further:

These lists are designed to be used in hosts format. That format, by definition, is made for things like /etc/hosts. Putting ANY domain in that format will result in an IP address being returned. /etc/hosts has no method to respond in any other fashion.

If this domain is added in to a list then Pi-hole will (possibly) allow any client using FF to use DoH, this will be immediately noticed by any user as they will lose ad blocking. This is a great teaching moment. Users learn that they need to use reputable lists (and the minimum number of them possible) and they need to maintain their DNS servers and not just install and forget. This isn't plug and play nor should it be.

Screenshot 2021-04-10 103552

Found the discussion on this:

I think the gist of the discussion was that whitelisting use-application-dns.net should preclude any list provider from erroneously adding the domain. My quick tests shows that works:

dan@raspberrypi:~ $ pihole -q -exact use-application-dns.net
  [i] No exact results found for use-application-dns.net within the block lists

dan@raspberrypi:~ $ dig use-application-dns.net @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> use-application-dns.net @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 13841
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

dan@raspberrypi:~ $ pihole -w use-application-dns.net
  [i] Adding use-application-dns.net to the whitelist...
  [✓] Reloading DNS lists

dan@raspberrypi:~ $ dig use-application-dns.net @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> use-application-dns.net @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 9818
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

dan@raspberrypi:~ $ pihole -q -exact use-application-dns.net
 Exact match found in exact whitelist
   use-application-dns.net

@jpgpi250 Can you confirm this behavior or find a situation that this would not work?

added exact whitelist entry for use-application-dns.net, dig returns NXDOMAIN, thus solution works.

Still implies the users must add the whitelist entry to neutralize blocklist entries for the domain use-application-dns.net, this to ensure firefox will NOT use DoH.

Yes, and it could be better documented.

I agree we should do something about it to improve the "out-of-the-box behavior" especially for newcomers that don't want to dig through any documentation before everything works. And it is a very special and interesting issue @jpgpi250 brought up. Looking at the available options, the only viable way seem to be also the simplest:

After having finished the gravity run (database is ready-to-use), we go and delete the canary domain from the list (if exists). Two reasons:

  1. If the canary domain in in there, this will lead to problems. The average user won't expect this. They don't see why a blocked domain can make Firefox use DoH.
  2. This can be done very simple and quickly (less than 1 msec on my Pi) when we just finished building the tree. No need for any comparing every domain at insertion time nor any extra infrastructure like a new table for FTL.
2 Likes

Just received a github notification the domain has been removed from the list. This doesn't make your proposed code change obsolete, the domain may appear on other lists, now, or in the future...

Excellent stuff!

I had made the same observation a while ago (occurred with another list), and had decided just to whitelist the canary domain. This solution is a lot better as it indeed doesn't require user to 'dig'.

Way to go!

Here's the problem I see with the proposed solution.

It makes Pi-hole the arbiter of the contents of adlists.

We've repeatedly told the community and people that request we remove their domains from adlists the same thing: "Pi-hole does not create, curate, or modify adlists." I don't want to go down that path. What makes this domain any different than say, opendns.com or github.com or pi-hole.net. Why should we override users selected adlists?

I think the better solution, and one that helps not only Pi-hole but adlist maintainers and users of other applications or just plain /etc/hosts users, is to ask FireFox why 0.0.0.0 isn't considered to be a sentinel value to signal a condition. In this lists case it was intended for use-application-dns.net to return 0.0.0.0 and that should be respected.

I remember there was some conversation over this being possible when the canary domain was first proposed, it may already be in the code but just not documented, I haven't checked.

1 Like

https://searchfox.org/mozilla-central/source/browser/components/doh/DoHHeuristics.jsm#185

async function globalCanary() {
  let { addresses, err } = await dnsLookup(GLOBAL_CANARY);

  if (err === NXDOMAIN_ERR || !addresses.length) {
    return "disable_doh";
  }

  return "enable_doh";
}

Would need to add a check for 0.0.0.0 in the addresses array. (Or 0.0.0.0|:: if you want IPv6).

I don't want to get in a fight over this, I've already have, and recommend the whitelist solution (provided by you) to everybody who is reading this, unfortunately, lots of users don't read this (current read count 89).
You stated earlier you consider this to be a great teaching moment, and, for some users, that really want to dig deep into pi-hole and DNS, it actually is. There are however a lot of users who just want the service to work, no more, no less, and get annoyed when they have to troubleshoot the service again...
My main argument, pro DL6ER's code is that, unlike use-application-dns.net, opendns.com, github.com and pi-hole.net actually serve content. By changing / adding lists to block these three domains (examples) , you're actually blocking content, the user can no longer access. I fully agree this is not something you want to do, this to avoid complaints, topics, that question that decision.
Ensuring use-application-dns.net returns NXDOMAIN, doesn't block anything (no content), it just ensures the domains that the user added ,using blocklist(s), will not work (show content) when using firefox, which was the original intention of adding the setting to 01-pihole.conf (emergency release of latest pihole v4 version and the pihole v5 beta).

I was actually thinking about why we have that domain in the first place. I'm open to the argument that treating Firefox special shouldn't have been done in the first place.

It was a knee-jerk reaction to their decision to force users to opt-out of DoH instead of opt-in and I think we've actually inadvertently assisted in the proliferation of DoH.

This discussion is going to happen. It could happen here, it could happen on GitHub in the PR or it could happen privately between the developers. I thought it best to have the discussion in the open in the place where it was already being discussed to keep from fragmenting the conversation. If you want to ignore the further discussion between the developers, feel free.

I agree, but I think that annoyance should be directed at FireFox for their decision and not at us trying to patch over their decision.

For me the solution provided by DL6ER is absolutely ok.
use-application-dns.net is a special case and not comparable with domains visited by users in daily surfing.

Wouldn't the proper solution be one that fixes the problem for everybody? Again, we're just patching things and not actually solving the problem.

Advocate of the devil...

Isn't pihole just a patch that tries to fix annoyances on the internet, like ads, malware, and so many other stuff, people don't want to see?

True, but unfortunately, opting out of a lot of things is a reality in todays world, not just DoH. For example, My WIFI acces point is xxxxxx_optout_nomap, this to (hopefully) prevent Microsoft and google to map and locate my access point.

Given todays DoH reality, I would actually be very happy if more browsers, maybe even the OS, offered the option (canary domain), to ensure that system would never use DoH.

My hope still is SURICATA will sooner or later add rules to prevent DoH on my network. It has been suggested...

I guess, so then the problem should be solved by Firefox then since this is all about a choice that Firefox made. Just furthers my resolve that the whole canary domain should be removed from Pi-hole and let users see what Firefox decided to do.

Edit:

The canary domain is great, I think it can be improved and then everybody can use it. People that just want to add a single /etc/hosts entry, people that use Windows and can add an entry to their HOSTS file, people that use other adblockers....

This is a valid point. However, users will see "Pi-hole is not working". They will not see "Firefox enforces DoH". They will not see this because DoH is completely transparent. They don't know it exists so they cannot complain about it. However, they know Pi-hole exists and (apparently) isn't doing what they want: block ads.

Not handling the canary domain in the way Firefox is happy with would make Pi-hole stop working for the majority of users - the ones with Firefox. So it isn't something I'd like to consider.

My PR adds a patch for something we have forgotten: That blocked content takes precedence over dnsmasq configuration. This is a defect and my PR fixes this for this one special case we have.

Yes. But they have no real motivation to do this. There is no pressing need to, at least. With them not accepting 0.0.0.0 the worst that happens is that Pi-hole doesn't work. May not be all that high in priority for them because really nobody will come and complain about this.

In contrast, whenever such this domain pops up in a adlist, the frustration of Pi-hole users may grow because they cannot find out what is going wrong (again, because they may not even know DoH and its opt-out is even a thing). This leads to frustration and an increase in support load for our team. Both can be easily fixed with my PR completing the canary domain handling in the edge case we have overlooked so far.

To signal that their local DNS resolver implements special features that make the network unsuitable for DoH, network administrators may configure their networks to modify DNS requests for the following special-purpose domain called a canary domain : use-application-dns.net .

[...]

The result will be considered positive if:

  • The query completes with NOERROR and contains A or AAAA records (or both)

A negative result will be a signal to disable application DNS, i.e. DoH.
(source)

A 0.0.0.0 is still an A record and, hence, it gets blocked. Think also about other blocking modes Pi-hole offers, say IP blocking. They may even server a real IP address like 192.168.0.10. Hence, we should exclude this domain from being added to gravity. Because it is the simplest, fastest and most reliable way to achieve what we want.

In the end, it is about us wanting that Pi-hole works out of the box with Firefox. It is not Firefox wanting to work best with Pi-hole. So they will not become active but we should.

1 Like