Update: it seems to be related to DNSSEC. Once I disabled it, streaming podcasts immediately worked.
I use the following upstream DNS servers:
- Quad9 (filtered, DNSSEC)
- Custom: 46.182.19.48
How can I track down which one is the problematic one?
Only use one and see if the error persists? If it does, only use the other one and see if the error is gone?
Sure, but what next:
- Why is it broken suddenly?
- Is the „avoid the problematic one“ approach sufficient?
I also meanwhile noted that downloading/streaming some podcasts in the official Podcasts app from iOS is broken as well.
Seems like one or more CDNs are affected. DNSSEC… for what it‘s worth -.-
Doesn’t really matter which one is used. Tried all combinations (only use the one, only use the other, use both, use none of them).
Even updating the custom one to a new IP according to Zensurfreier DNS-Server | Digitalcourage made no difference.
Once I disable the DNSSEC setting, it immediately works (and e. g. a blocked podcast download in the Apple Podcasts app starts instantly).
- Is changing DNS upstream server settings in the UI and force-closing the affected apps to trigger a new name resolution sufficient?
- Or do I need to manually do a „pihole restartdns“ after every upstream server change?
I suspect there‘s some kind of cache involved here or it’s a Pi-Hole internal issue.
I also noticed that after every upstream DNS server change (hit „Save“) Pi-Hole went offline after a few seconds and I needed to manually start it again. If that‘s normal, fine.
That should not happen. It should restart fine from alone.
Please upload a debug log and post just the token URL that is generated after the log is uploaded by running the following command from the Pi-hole host terminal:
pihole -d
or do it through the Web interface:
Tools > Generate Debug Log
- The debug log contains (way too) much personal information.
- I could observe this behaviour (Pi-Hole offline when making changes to upstream DNS server changes) only once out of 30 to 40 changes - I don't really mind.
- Back to the original issue: Spotify being completely dead when DNSSEC is active. I tried so much for almost 2 hours:
- tested every upstream dns server one by one --> it's not depending on the server selection
- i closely monitored the hostnames in query log when trying to play/download Spotify content (which has not yet been cached). only blocked ones are
2022-04-08 23:48:37 A firebaselogging-pa.googleapis.com iPhone Blocked (gravity) IP (0.0ms)
2022-04-08 23:48:37 HTTPS firebaselogging-pa.googleapis.com iPhone Blocked (gravity) NODATA (0.1ms)
- turning Pi-Hole off and trying to play Spotify podcast still does not work
- Once I disable DNSSEC (even with Pi-Hole blocking still OFF
) playing Spotify podcasts immediately works❗
So what to do with that information?
It's not a blocking thing, it doesn't depend on the upstream server, it's a DNSSEC thing. Going crazy!
- What happens between a client, Pi-Hole and "the internet" once Pi-Hole is being disabled: are client requests bypassed and sent to (where)? Is the Pi-Hole cache used?
- Because: if disabling DNSSEC at the time Pi-Hole is off makes my client being able to access content, Pi-Hole has to be involved in some way.
I don't make any more progress here, professionals help appreciated.
History: I also checked my change log. Back in december 2019 I had DNSSEC activated but because of similar (much worse) issues like not even being able to surf wikipedia.org I disabled it in February 2020. Some time later I reenabled DNSSEC (late 2020 or early 2021) and it worked without any issues - until a few weeks ago.
Likely not related to your observation, but note that resolution of the same domain may produce different results if you use upstream servers that each would employ different filtering strategies (i.e for a domain that is allowed by Pi-hole and forwarded upstream, Quad9 may block a domain that digitalcourage would not).
This is happening independently from your observation, posing a separate issue.
Since switching off DNSSEC allows you to resolve, regardless of upstream servers, that would suggest an issue with the configuration of DNS servers authoritative for the domains you cannot access when DNSSEC is enabled.
Have a look whether the DNSSEC related replies from your logs would hint at resolution denials (BOGUS) due to failed DNSSEC validation, see also Understanding DNSSEC validation using Pi-hole's Query Log).
This is why we request your debug log. Only a few members of the Pi-hole team have access to your log and it auto-expires in 48 hours.
Yes, likely not related. As I stated I was using other servers too like only Google DNS. Google + DNSSEC = issue, Google without DNSSEC = working. For me enough proof the selection of upstream server(s) is irrelevant here.
Checking the query log for the moment trying to access the content (stream Spotify podcast) with DNSSEC enabled only gives plenty of
and
.
Out of 154.443 entries in /var/log/pihole.log only 23 lines contain BOGUS - and all those requests are not related to the issue (other time slots).
If there would be a way to manually remove few details from the debug log before uploading I'm willing to provide it. Some parts are very privacy related (internal network structure, internal hostnames, admin mail adress, some whitelisted hostnames etc.) and they do provide little to no benefit for debugging. I know other FOSS projects like Nextcloud where such critical parts are anonymized/masked before uploading (e. g. ADMIN_EMAIL=******@******.***).
From this list, all but the email address are helpful to us in troubleshooting.
So how can I provide a filtered version of the debug log file? Via PN like I did with DL6ER some time ago?
Yes. PM me your log.
Edit - intact but with your email address removed.
Done.
If you'd switch off Pi-hole, you would have no DNS resolution at all (unless your router would cater for an alternative DNS server somehow).
I think you mean that you use Pi-hole's web UI to Disable Pi-hole:
Disabling Pi-hole won't cut DNS resolution; it will just stop Pi-hole from filtering requests. The behaviour is the same as running Pi-hole completely void of blocklists, white- and blacklists.
INSECURE results mean that Pi-hole could not employ DNSSEC to validate the domain because required respective DNSSEC related records are absent from somewhere in the validation path, i.e. the domain does not support DNSSEC validation.
This does not affect DNS resolution, Pi-hole will still supply the requested DNS records to the client.
Beyond the current log, you may search all available logs for BOGUS answers by running:
zgrep -n BOGUS /var/log/pihole.log*
It currently seems that you cannot correlate blocked or BOGUS replies with your client's inability to stream from Spotify.
As you observe no matching BOGUS replies, that would make an issue with a misconfigured authoritative DNS server less likely.
Your debug log contains quite a few instances of the following lines:
*** [ DIAGNOSING ]: contents of /var/log
-rw-r--r-- 1 pihole pihole 129K Apr 9 15:45 /var/log/pihole-FTL.log
-----head of pihole-FTL.log------
[2022-04-09 00:00:20.500 26610M] ERR: Port mismatch for 9.9.9.9: we derived 53, dnsmasq told us 48
[2022-04-09 00:00:20.500 26610M] ERR: Port mismatch for 5.9.164.112: we derived 53, dnsmasq told us 48
This same symptom has been reported a while ago, and it's woth noting it has some ties to DNSSEC.
Past analysis then concluded that while the message was logged, Pi-hole's DNS resolution would still be operational, as pihole-FTL can still acquire the correct port.
Nevertheless, that the Pi-hole team developed a fix for that issue (but also, dnsmasq would have to do some patching for its part).
You could try to use Pi-hole's fix and see if that would have an impact on your issue:
BUT BEFORE DOING SO:
Could you please extract sample portions of your /var/log/pihole.log as well as corresponding /var/log/pihole-FTL.log for the time when you fail to stream from Spotify?
This may help Pi-hole development to check if there would be any abnormalities beyond the Port mismatch warnings that potentially would explain your observation.
If you prefer, you may upload your logfile portions to Pi-hole's tricorder by running e.g. (assuming you've stored your excerpt to a file pihole.partial.log):
cat pihole.partial.log | pihole tricorder
Then provide us with the respective tokens as shown once the upload completes.
And finally, and again unrelated to your observation: (click for details)
From your debug log
-rw-r--r-- 1 root root 1,2K Apr 2 13:04 /etc/pihole/pihole-FTL.conf
; Am 2017-06-19 manuell erstellt und Konfiguration gem. https://github.com/pi-hole/FTL#ftls-config-file vorgenommen/angepasst
TIMEFRAME=rolling24h
That option is long deprecated and has been removed from Pi-hole in January 2018. Pi-hole's rolling 24 hour timeframe has become the fixed, unchangeable default.
Yes, I did so (thanks for the tricorder command, that makes things very easy/handy 
)
- https://tricorder.pi-hole.net/87Lh2imP/
- As noted in the log: nothing in pihole-FTL.log during the access time slot
==> Update: same attempt with DNSSEC disabled, see https://tricorder.pi-hole.net/IpFxhKer/ - can you see any difference?
New information:
- Doing the same thing on an Android device (also official Spotify app) works as intended without any issues
- Unfortunately I don't have another iDevice to figure out if it's
a) an iOS Spotify app DNS issue
b) an iDevice specific issue
==> Update: I tried to access/play the Spotify content from the browser on the iDevice, with the same outcome. So it's not an "app only" issue. - I don't use private relay etc. from Apple, only for mails so everything should flow through Pi-Hole
Thank you for the notice, I finally removed that old stuff from my early Pi-Hole days.
I'm testing this patch now, turning back shortly.
Tested, observations:
- after running that command,
pihole-FTL.logat the end gave
[2022-04-10 00:31:22.806 27337M] Reloading DNS cache
[2022-04-10 00:31:22.874 27337/T27341] Compiled 1 whitelist and 3 blacklist regex filters for 22 clients in 64.4 msec
[2022-04-10 00:31:22.971 27337M] Received: Real-Time Signal 0 (34 -> 0)
[2022-04-10 00:31:23.929 27337/T27341] Compiled 1 whitelist and 3 blacklist regex filters for 22 clients in 53.2 msec
[2022-04-10 00:31:24.809 27337M] Blocking status is disabled
and pihole status was
[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)
[✗] Pi-hole blocking is disabled
so I had to pihole enable. In case this is normal behavior, don't mind.
Back to topic: the patch changed nothing for my Spotify iDevice issue. Had it applied for a few minutes where no further entries in pihole-FTL.log popped up. Reverted back now with pihole checkout master (which, again, resulted in [✗] Pi-hole blocking is disabled by the way).
I have a new service not working on iDevice only. It is a bit easier to test and debug cause it’s a simple URL (for the Spotify case I still even don’t know what hostnames are responsible): https://bikers-voice.at
This time I could test with three devices:
- My iDevice: not loading
- My Android device: loading
- Another iDevice: not loading
What the heck is different with the fruits?
This one makes much more sense. www.bikers-voice.at returns BOGUS, even though bikers-voice.at is merely INSECURE. Even typing in https://bikers-voice.at my iPad refuses to connect.
So... what next? I provided everything and beyond, what else can I do to help you helping me on this?
Official app would still mean two separate pieces of software on iOS and Android, so their respective requested DNS resolutions may differ.
Do you see any differences in DNS requests from those two Spotify apps?
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.