How do I block ads on YouTube?

blacklisting

#20

I’m guessing the googlevideo.com domain is used for the videos themselves, and then the fingerprints subdomains are used for the ads. We might still be able to use the script to get the fingerprints and the wildcard the rest of the subdomains.

This is, of course, assuming @Excelerate246’s findings are true, but I think it’s a step in the right direction.


#21

From what i can tell the pi could never handle that amount of entries.

I also found that youtube connect to 3 things on the “googlevideo.com” domain:

manifest.googlevideo.com
redirector.googlevideo.com
and then the ones with the fingerprint.

But the white list does not seem to work on wildcard entries. So i cannot seem to exclude those entries to test.


#22

Yes. Due to the way wildcards work it is not possible to whitelist any domains (the wildcard will always take preference in the DNS server backend!).


#23

Would there not be any way to add a regular expression to the lists? This would make it very easy to add any kind of crap google seems to try here.


#24

No, the way DNS works is the limiter here. Let’s oversimplify things a bit, so it gets more obvious why wildcard blacklisting does what it is doing: Let’s assume we want to visit awesome.domain.de.

  1. The root servers will be asked if they know .de and us an address for a server that knows all domains that are provided under .de

  2. The .de server will know the host name and will return the address of the server that manages domain.de

  3. This server will eventually be asked for the address of awesome.domain.de and will give us the final address to which we will connect.

You see, DNS works from right to left. If we now wildcard block domain.de none of the above steps will happen. Instead, the Pi-hole will immediately answer its own IP (regardless of the subdomain).

I know that this might be inconvenient but rest assured that we had some sleepless nights, scratching our heads how to make it better and there seems to be no better way with the DNS resolver dnsmasq which we are using.


#25

Thanks for the information.

Using this post: How do I add Wildcard sites to the blacklist?

I was able to white list the mentioned domains, and the tail log shows it’s going trough, but there is still no video.
Also, using this method still shows it as piholed in the query log. But the tail log shows it going trough.
This leads me to believe that this wont work.


#26

Well, Yes and No. Let me explain:

Yes, indeed, this method works. However, we did not implement it in the way you added it now, since (due to the right-to-left nature) this is also a wildcard whitelisting.

Say you wildcard blocklist domain.de and wildcard whitelist something.domain.de. While e.g. trash.domain.de will still be blocked, some.other.ad.something.domain.de will be permitted. This might be unexpected.

No, it shows it because it matches the wildcard blacklist filter. The wildcard whitelisting is not (officially) supported and hence the filter does not know about it.


#27

This could indeed give problems, but for testing this instance it would white list what i needed, so the test is still valid.
It blocked all calls to any of the “fingerprinted” domains and let the 2 aforementioned domains trough, still no video. This means that youtube does not only send it’s adds via those domains, but also the content. So blacklisting it, with a wildcard or otherwise, is not our solution to blocking these adds.


#28

This is a greatly appreciated result!


#29

Isn’t it possible to make the name itself accept wildcards? So then we don’t wildcard a whole domain.
What i mean is this:
Current situation is to wildcard blacklist example.domain so everything that has that domain is blacklisted.

What i hope we can achieve is this: some*.example.domain where the * is the wildcard. So everything like the following domains are blacklisted:
-somestart.example.domain
-something.example.domain
-someinsert.example.domain

but helliamallowed.example.domain is allowed and not blacklisted?

I hope the point i’m hoping to make is clear enough. I think it must be implemented by a sort of mask like an ipv4 mask. everything that hits the mask is blacklisted and everything thats not is allowed through.


#30

Yes, I understand your request and would be really happy to do this. However, you have to keep in mind that we use the DNS resolver dnsmasq in the backend. This service is not capable of doing like you suggested. Therefore, we can not offer this to our users. Even if we would find the time to implement this in dnsmasq (which is close to impossibel with the current workload of us developers) we would still have to wait quite some long time until the new version of dnsmasq is shipped with the distributions package systems.


#31

I understand yes, I don’t know much about dnsmasq so thats to bad :frowning:

Thanks for replying and taking the time to explain. And thanks for developing this, its awesome :slight_smile:


#33

I have checked https://dnsdumpster.com and it shows few domains related to googlevideo.com. Does it changes each day?


#34

Like i posted before, the content AND the adds come from the same domains.
So blocking these domains is not a good idea.
You wil end up blocking both the adds and what you want to see.

Sadly this is not the solution to YouTube adds. Still looking for more, but pretty sure we cant block YouTube adds via domain.


#35

I have to admit that I never see any ads on Youtube. I cannot tell what might be the reason, but I guess it is a mixture of my conservative use of Youtube (mostly only classical music videos) and the interval I watch videos (less than one per day, usually). I let the movies run in the background but I’d hear if there would be some ads content.

I heard people talking about some “yellow stripes” on the timeline but I clicked through a couple of videos (even fairly long ones) and have not been able to find such stripes, e.g.

Although there is always such a yellow stripe at the beginning of the timeline, I never see any ads at the beginning of the movie.

More details on my local setup if you are interested:

  • Up-to-date Pi-hole with only stock lists
  • Up-to-date Linux Mint + up-to-date Chrome browser
  • No adblocking related browser plugins (I use Zotero Connector, ZenHub for GitHub, RemoveCookieFromSite and Session Buddy).

#36

Any further developments?


#38

According to the above reply this might not work.
However trough my (short!) tests I haven’t seen any ads whilst using pi-hole to block the googlevideo.com subdomains.

Anyhow, to automate a lot of the manual steps I made a script.

So you will need the unoffical dns dumpster API from here: https://github.com/PaulSec/API-dnsdumpster.com
I cloned this git to a folder /etc/dnsdumpster/ with the commands:

mkdir /etc/dnsdumpster
git clone https://github.com/PaulSec/API-dnsdumpster.com /etc/dnsdumpster

If everything went according to plan you should now have 4 files in the above folder amongs which should be:

/etc/dnsdumpster/API-example.py
/etc/dnsdumpster/DNSDumpsterAPI.py

Make sure you have python and pip installed and use the following commands:

cd /etc/dnsdumpster
pip install -r requirements.txt

Now edit the API-example.py using vi
you have to edit the line

res = DNSDumpsterAPI(False).search('tesbro.com')

to the following

res = DNSDumpsterAPI(False).search('googlevideo.com')

Now this should make the example api output everything that dnsdumpster shows/does

Next is to make a script in /etc/pihole, I named it youtube-ads.sh
It should contain the following code:

rm /etc/dnsdumpster/youtube-domains.txt
rm /etc/dnsdumpster/youtube-filtered.txt
rm /etc/dnsdumpster/youtube-ads.txt
python /etc/dnsdumpster/API_example.py > /etc/dnsdumpster/youtube-domains.txt
grep "^r" /etc/dnsdumpster/youtube-domains.txt > /etc/dnsdumpster/youtube-filtered.txt
sed 's/\s.*$//' /etc/dnsdumpster/youtube-filtered.txt > /etc/dnsdumpster/youtube-ads.txt
cp /etc/dnsdumpster/youtube-ads.txt /var/www/html/youtube.txt
pihole -g

Make the script executable:

chmod +x /etc/pihole/youtube-ads.sh	

Now you should do the following to add your youtube.txt to the block lists in pihole:

cp /etc/pihole/adlists.default /etc/pihole/adlists.list
vi /etc/pihole/adlists.list

In this file add the following to the bottom:

# My local list
http://localhost/youtube.txt

Save the file and edit your crontabs to automate the process

crontab -e

Add the following line:

0 0 * * * /etc/pihole/youtube-ads.sh

This should run the script daily and might remove youtube ads.

– I am by no means a linux/pi-hole expert, the above is what I did to make it work.
Sure things could be better and if anyone has anything to add, please do tell me!
All I know is that for me it seems to work using the above.


#39

I added to your bash script – also modified the line where you ‘cp’ and instead append – with your method, it overwrites the file after each run

rm /etc/dnsdumpster/youtube-domains.txt
rm /etc/dnsdumpster/youtube-filtered.txt
rm /etc/dnsdumpster/youtube-ads.txt
python /etc/dnsdumpster/API_example.py > /etc/dnsdumpster/youtube-domains.txt
grep "^r" /etc/dnsdumpster/youtube-domains.txt > /etc/dnsdumpster/youtube-filtered.txt
sed 's/\s.*$//' /etc/dnsdumpster/youtube-filtered.txt > /etc/dnsdumpster/youtube-ads.txt
cat /etc/dnsdumpster/youtube-ads.txt >> /var/www/html/youtube.txt
#greps the log for youtube ads and appends to /var/www/html/youtube.txt
grep r*.googlevideo.com /var/log/pihole.log | awk '{print $6}'| grep -v '^googlevideo.com\|redirector' | sort -nr | uniq >> /var/www/html/youtube.txt
#removes duplicate lines from /var/www/html/youtube.txt
perl -i -ne 'print if ! $x{$_}++' /var/www/html/youtube.txt
#updates pihole blacklist/whitelist
pihole -g

My additions scans the ‘pihole’ logs and pulls ones that may be missed. Then it removes duplicates from the file ‘/var/www/html/youtube.txt’

…I also changed the ‘cron’ to run every 5 minutes so it grabs more.

…It’s definitely ugly looking but it works…


#40

That does not seem to be a good idea, depending on the hardware you are using that means that your Pi-hole will not respond for several seconds every 5 minutes. Might annoy users on your local network.


#41

Ah you’re correct – the DNS service takes about a millisecond blip running on my RPi3 – which is also a RetroPie. I ran these 2 commands simultaneously:

# time pihole -g
# for i in {1..1000} ; do pihole status; done | tee /root/test.txt

I’ve tested a few times and have only been able to catch “DNS service is stopped” once in the logs through my 1000 status checks. The overall task of running ‘pihole -g’ takes about 16 seconds on my device. I did check the bash script that is called when ‘pihole -g’ is run and it’s located at ‘/opt/pihole/gravity.sh’ by default on my device. It looks like the only call that would affect service is ‘systemctl restart dnsmasq’. I may be wrong.

I’m going on <24 hours of using Pi-Hole so I’m in no way an expert – my “Upstream DNS” is listed as 8.8.8.8 and my Router’s primary DNS is the Pi-Hole but the Secondary is 8.8.8.8 so I believe either way I’m covered as far as DNS…

Nonetheless, DL6ER, I took your advice and changed the ‘cron’ to run every 6 hours instead of 5 min. Thanks!