pihole-FTL (NOT sure there is a problem - requesting clarification)

General Beta 5.0
1

I'm currently analizing this (youtube ad blocking) script, the result is raising some questions.

1. I'm a bit confused about the pihole restartdns command.
According to the help

  restartdns          Full restart Pi-hole subsystems
                        Add 'reload' to update the lists and flush the cache without restarting the DNS server
                        Add 'reload-lists' to only update the lists WITHOUT flushing the cache or restarting the DNS server

pihole restartdns reload-lists should update the lists, however if I add entries to a file (/etc/hosts.youtube) and execute that command, I still get the real IP address, as opposed to the configured address (using dig or nslookup).
When I use pihole restartdns reload, the configured IP address is returned.

The help entry says reload-lists should update the lists but apparently, only reload does that.

When I use reload-lists, there are no pihole.log entries, when I use reload, the following messages appear in the log (which is what I want, but expected to happen using reload-lists:

May  8 10:04:45 dnsmasq[12831]: read /etc/hosts - 15 addresses
May  8 10:04:45 dnsmasq[12831]: read /etc/hosts.youtube - 166 addresses
May  8 10:04:45 dnsmasq[12831]: read /etc/localdns.list - 40 addresses
May  8 10:04:45 dnsmasq[12831]: read /etc/pihole/custom.list - 0 addresses
May  8 10:04:45 dnsmasq[12831]: read /etc/pihole/local.list - 4 addresses

What am I missing?

2. cname blocking
The script from the topic, mentioned earlier, ads entries to /etc/hosts.youtube. I've already changed /usr/local/bin/pihole restartdns reload-lists into /usr/local/bin/pihole restartdns reload, to overcome the first problem (see above)

The script ads the following entries (example):

173.194.182.198 r5.sn-4g5e6nzs.googlevideo.com

This is an entry for the A record, dig result:

pi@raspberrypi:~ $ dig +noidnin +noidnout r5---sn-4g5e6nzs.googlevideo.com

; <<>> DiG 9.11.5-P4-5.1-Raspbian <<>> +noidnin +noidnout r5---sn-4g5e6nzs.googlevideo.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29317
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;r5---sn-4g5e6nzs.googlevideo.com. IN   A

;; ANSWER SECTION:
r5---sn-4g5e6nzs.googlevideo.com. 30 IN CNAME   r5.sn-4g5e6nzs.googlevideo.com.
r5.sn-4g5e6nzs.googlevideo.com. 30 IN   A       74.125.13.234

;; Query time: 3 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri May 08 10:20:12 CEST 2020
;; MSG SIZE  rcvd: 106

When you start a youtube video, a request is made to r5---sn-4g5e6nzs.googlevideo.com, which is a cname for r5.sn-4g5e6nzs.googlevideo.com
since r5.sn-4g5e6nzs.googlevideo.com is in /etc/hosts.youtube, I hoped the cname blocking feature would also notice there is a configured address for the A record, but apparently, the cname feature doesn't do that. Is this intentional (works as designed - block only) or is there something I'm missing?

Thank you for your clarifications.

2

Do you have 5---sn-4g5e6nzs.googlevideo.com on an adlist/blocklist or why is it blocked?

3

You are right, the description is somewhat inexact, we will change this (however, it will probably go into v5.1 now):

  1. restartdns = restarts the entire FTL, incl. config file parsing and history re-reading
  2. restartdns reload = reload the DNS cache (this implies reloading the HOSTS files as you've seen) + reload database tables (white-/blacklist, gravity, groups, clients)
  3. restartdns reload-lists = only reload database tables (white-/blacklist, gravity, groups, clients), the embedded dnsmasq core doesn't see anything from this signal

CNAME blocking is quite hefty in its performance hit. Due to technical reasons, the CNAME inspection cannot use results from the DNS cache. For one, because the CNAME blocking could be spoiled easily and behave in a way users don't expect and which is hard to explain. For the other part, successfl lookups in dnsmasq's cache immediately trigger a response to the client (which is what we want to avoid!). As the cache also contains all HOSTS file knowledge in dnsmasq, we cannot easily access this.
Hence, CNAME blocking can only respect entries from the black-/whitelist and gravity (everything what is in the gravity.db database) but not the HOSTS files. This cannot even be changed easily so it may be considered intentional.

4

Thank you for this (glad it turns out I'm not loosing my mind - yet), apart from the incorrect message (pihole command), everything is working as you expected.

The script will need to be modified by the author (@grublets), as, given your explanation, it currently doesn't do anything, unless your restart or reload pihole-FTL.

5

The easiest solution might be to add the HOSTS file containing the youtube domains as adlist (using the file:// "protocol"). By this, you will also be able to manage who is getting this blocked through groups.

6

Does pihole respect IP -> domain configurations stored in the HOSTS file loaded as adlist or does it only take the domains and applies the configured blocking?
This hostfile looks like this

89.27.131.204 r1.sn-4g5e6nsd.googlevideo.com
89.27.131.204 r1.sn-4g5e6nzl.googlevideo.com
89.27.131.204 r1.sn-4g5ednly.googlevideo.com
89.27.131.204 r1.sn-4g5edns6.googlevideo.com
89.27.131.204 r1.sn-4g5ednsr.googlevideo.com
89.27.131.204 r1.sn-8xgn5uxa-cxge.googlevideo.com
89.27.131.204 r1.sn-8xgn5uxa-cxgl.googlevideo.com
89.27.131.204 r1.sn-8xgn5uxa-quhs.googlevideo.com
89.27.131.204 r1.sn-qxo7sney.googlevideo.com
89.27.131.204 r1.sn-qxoedn7z.googlevideo.com
89.27.131.204 r2.sn-4g5e6nls.googlevideo.com
89.27.131.204 r2.sn-4g5e6ns6.googlevideo.com
89.27.131.204 r2.sn-4g5edne6.googlevideo.com
89.27.131.204 r2.sn-4g5edns6.googlevideo.com

and as far as I have understood this it is not intended to block those domains but to redirect them.

7

Don't think that's an option

  1. adding it as an adlists would result in the domain being blocked (0.0.0.0). The goal is to change the IP address into an alternate address, not to block it.
  2. the author advises to run cron every minute or so. Your suggestion would imply running pihole -g every time there is a change -> NOT an option....

That's what I think, as usual, I may be wrong...

8

I see, you are right, I forgot about that for a second.

9

If adding both the A record and the CNAME to the the hosts file is helpful I believe the script owner would just have to change the following from:

cp $ytHosts $workFile
zgrep -e "reply.*-.*\.googlevideo.*\..*\..*\..*" $piLogs \
    | awk -v fIP=$forceIP '{ print fIP, $6 }' >> $workFile

to:

cp $ytHosts $workFile
zgrep -e "reply.*-.*\.googlevideo.*" $piLogs \
    | awk -v fIP=$forceIP '{ print fIP, $6 }' >> $workFile

I know this is not the same has having full CNAME lookup.

12

adding both r5---sn-uxaxoxu-cg0k.googlevideo.com AND r5.sn-uxaxoxu-cg0k.googlevideo.com entries eventually leads to An error occurred , no more video's. Originally (the current script), only ads the r5.sn-uxaxoxu-cg0k.googlevideo.com (A record) entry, tailing the pihole log shows youtube always requests the r5---sn-uxaxoxu-cg0k.googlevideo.com (CNAME) entry

13

This issue is solved, use Issues · grub / YouTube updater for Pi-hole · GitLab or the topic for this script for further conversation.