Use-application-dns.net found in adlists

Last week, I noticed use-application-dns.net is on a blocklist.
When firefox does the query, to check if it can use DoH, the reply is 0.0.0.0 (BLOCKINGMODE=NULL), as opposed to the required NXDOMAIN reply. The blocklist entry appears to have precedence over the 'server=/use-application-dns.net/' setting, that was added last year to /etc/dnsmasq.d/01-pihole.conf.

I know the instant reaction will be 'do NOT use sh**ty blocklists', and I fully support this point of view, however, you cannot expect all users to check whether 'use-application-dns.net' has been added to a blocklist they happened to use, after every (sunday automatic) gravity run.

There are multiple options to ensure this domain doesn't end up in the gravity database:

  • ensure the domain isn't included, when the database is build. Probably NOT the best idea, since every new entry would need to be compared to this domain (and hopefully / possibly additional canary domains in the future)
  • Make FTL smart enough to respond with NXDOMAIN for entries in a new database table, containing these entries. The entries in this table could be pre-populated (pihole-FTL creates the database, if it doesn't exist), no web interface required, so the domain entries would remain under control of the developers.
  • Create a new database table, containing the NXDOMAIN(s), again, can be prepopulated, and remove the entries in this table from the gravity table, after the new temp gravity database has been fully populated, before switching the database (tmp -> active). This method is probably the most efficient, at this time (only one canary domain), it would require only one additional sqlite3 'delete from' instruction.

I'm aware this will be a controversial request, but it may help users, that install an use pihole without ever looking back.

Did you alert the blocklist maintainer?

Assuming your observation would apply to other DNS filters as well, putting that canary domain on a blocklist would potentially ruin Firefox behaviour for those as well.

new issue, see here.

1 Like

What we have (and had to) consider is how to handle users that want the canary to resolve and enable the feature on their setups? From my recollection of the discussion, this behavior is exactly intended.

You can add this domain to a group if you wanted and have that group configured to use FF's DoH.

So, yes, what is happening is what should be happening.

Side note: The maintainers of the list in question are the same people that tried to sell Pi-hole devices and then rebranded when their scammy behavior came to light. I think they also allow companies to "sponsor" their lists?

1 Like

Apologies, I don't understand what you're saying.

  • pihole development added ' server=/use-application-dns.net/ ' to /etc/dnsmasq.d/01-pihole.conf to ensure a query (from firefox) got the mandatory NXDOMAIN reply, this to ensure firefox would NOT bypass pihole, by using DoH.
  • whatever construction you setup with groups or other gravity methods, while using the default blocking mode (NULL), the reply for use-application-dns.net will always be 0.0.0.0, which would than allow firefox to use DoH, making having a pihole useless (when using firefox).
    Are there really users that have pihole but don't want firefox to use it? If so, they can always add a dnsmasq custom config to ensure firefox is allowed to use DoH (or modify the default entry in /etc/dnsmasq.d/01-pihole.conf. They would have already made configuration changes to allow this, so probably knowledgeable users.

I've already taken precautions, to ensure the entry never makes it into gravity. Just trying to help and protect other users, that may not even be aware of OR notice a blocklist entry allows firefox to use DoH.

By default, with the ability to override our decision with what the user actually wants to happen.

Yes, this is the same as putting users in a group that get no filtering. Users have to have the ability to disable a function if they so desire.

Yes, and that is their choice.

Going a step further:

These lists are designed to be used in hosts format. That format, by definition, is made for things like /etc/hosts. Putting ANY domain in that format will result in an IP address being returned. /etc/hosts has no method to respond in any other fashion.

If this domain is added in to a list then Pi-hole will (possibly) allow any client using FF to use DoH, this will be immediately noticed by any user as they will lose ad blocking. This is a great teaching moment. Users learn that they need to use reputable lists (and the minimum number of them possible) and they need to maintain their DNS servers and not just install and forget. This isn't plug and play nor should it be.

Screenshot 2021-04-10 103552

Found the discussion on this:

I think the gist of the discussion was that whitelisting use-application-dns.net should preclude any list provider from erroneously adding the domain. My quick tests shows that works:

dan@raspberrypi:~ $ pihole -q -exact use-application-dns.net
  [i] No exact results found for use-application-dns.net within the block lists

dan@raspberrypi:~ $ dig use-application-dns.net @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> use-application-dns.net @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 13841
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

dan@raspberrypi:~ $ pihole -w use-application-dns.net
  [i] Adding use-application-dns.net to the whitelist...
  [✓] Reloading DNS lists

dan@raspberrypi:~ $ dig use-application-dns.net @127.0.0.1

; <<>> DiG 9.11.5-P4-5.1+deb10u3-Raspbian <<>> use-application-dns.net @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 9818
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

dan@raspberrypi:~ $ pihole -q -exact use-application-dns.net
 Exact match found in exact whitelist
   use-application-dns.net

@jpgpi250 Can you confirm this behavior or find a situation that this would not work?

added exact whitelist entry for use-application-dns.net, dig returns NXDOMAIN, thus solution works.

Still implies the users must add the whitelist entry to neutralize blocklist entries for the domain use-application-dns.net, this to ensure firefox will NOT use DoH.

Yes, and it could be better documented.

1 Like

I agree we should do something about it to improve the "out-of-the-box behavior" especially for newcomers that don't want to dig through any documentation before everything works. And it is a very special and interesting issue @jpgpi250 brought up. Looking at the available options, the only viable way seem to be also the simplest:

After having finished the gravity run (database is ready-to-use), we go and delete the canary domain from the list (if exists). Two reasons:

  1. If the canary domain in in there, this will lead to problems. The average user won't expect this. They don't see why a blocked domain can make Firefox use DoH.
  2. This can be done very simple and quickly (less than 1 msec on my Pi) when we just finished building the tree. No need for any comparing every domain at insertion time nor any extra infrastructure like a new table for FTL.
2 Likes

Just received a github notification the domain has been removed from the list. This doesn't make your proposed code change obsolete, the domain may appear on other lists, now, or in the future...

Excellent stuff!

I had made the same observation a while ago (occurred with another list), and had decided just to whitelist the canary domain. This solution is a lot better as it indeed doesn't require user to 'dig'.

Way to go!

Here's the problem I see with the proposed solution.

It makes Pi-hole the arbiter of the contents of adlists.

We've repeatedly told the community and people that request we remove their domains from adlists the same thing: "Pi-hole does not create, curate, or modify adlists." I don't want to go down that path. What makes this domain any different than say, opendns.com or github.com or pi-hole.net. Why should we override users selected adlists?

I think the better solution, and one that helps not only Pi-hole but adlist maintainers and users of other applications or just plain /etc/hosts users, is to ask FireFox why 0.0.0.0 isn't considered to be a sentinel value to signal a condition. In this lists case it was intended for use-application-dns.net to return 0.0.0.0 and that should be respected.

I remember there was some conversation over this being possible when the canary domain was first proposed, it may already be in the code but just not documented, I haven't checked.

1 Like

https://searchfox.org/mozilla-central/source/browser/components/doh/DoHHeuristics.jsm#185

async function globalCanary() {
  let { addresses, err } = await dnsLookup(GLOBAL_CANARY);

  if (err === NXDOMAIN_ERR || !addresses.length) {
    return "disable_doh";
  }

  return "enable_doh";
}

Would need to add a check for 0.0.0.0 in the addresses array. (Or 0.0.0.0|:: if you want IPv6).

1 Like

I don't want to get in a fight over this, I've already have, and recommend the whitelist solution (provided by you) to everybody who is reading this, unfortunately, lots of users don't read this (current read count 89).
You stated earlier you consider this to be a great teaching moment, and, for some users, that really want to dig deep into pi-hole and DNS, it actually is. There are however a lot of users who just want the service to work, no more, no less, and get annoyed when they have to troubleshoot the service again...
My main argument, pro DL6ER's code is that, unlike use-application-dns.net, opendns.com, github.com and pi-hole.net actually serve content. By changing / adding lists to block these three domains (examples) , you're actually blocking content, the user can no longer access. I fully agree this is not something you want to do, this to avoid complaints, topics, that question that decision.
Ensuring use-application-dns.net returns NXDOMAIN, doesn't block anything (no content), it just ensures the domains that the user added ,using blocklist(s), will not work (show content) when using firefox, which was the original intention of adding the setting to 01-pihole.conf (emergency release of latest pihole v4 version and the pihole v5 beta).

I was actually thinking about why we have that domain in the first place. I'm open to the argument that treating Firefox special shouldn't have been done in the first place.

It was a knee-jerk reaction to their decision to force users to opt-out of DoH instead of opt-in and I think we've actually inadvertently assisted in the proliferation of DoH.

This discussion is going to happen. It could happen here, it could happen on GitHub in the PR or it could happen privately between the developers. I thought it best to have the discussion in the open in the place where it was already being discussed to keep from fragmenting the conversation. If you want to ignore the further discussion between the developers, feel free.

I agree, but I think that annoyance should be directed at FireFox for their decision and not at us trying to patch over their decision.

For me the solution provided by DL6ER is absolutely ok.
use-application-dns.net is a special case and not comparable with domains visited by users in daily surfing.

Wouldn't the proper solution be one that fixes the problem for everybody? Again, we're just patching things and not actually solving the problem.

Advocate of the devil...

Isn't pihole just a patch that tries to fix annoyances on the internet, like ads, malware, and so many other stuff, people don't want to see?

True, but unfortunately, opting out of a lot of things is a reality in todays world, not just DoH. For example, My WIFI acces point is xxxxxx_optout_nomap, this to (hopefully) prevent Microsoft and google to map and locate my access point.

Given todays DoH reality, I would actually be very happy if more browsers, maybe even the OS, offered the option (canary domain), to ensure that system would never use DoH.

My hope still is SURICATA will sooner or later add rules to prevent DoH on my network. It has been suggested...