What would that be??
crontab
/35 * * * * /etc/pihole/temp.sh
0/3 * * * * /etc/pihole/youtube-ads.sh
This syntax is wrong. it's either */35 or just 35.
Here is a crontab generator
https://crontab-generator.org/
crontab repeats a command with a periodicity
*/3 */5 * * * /etc/pihole/temp.sh
that means repeat command every 3 minutes, every 5 hours, every day,every month, every weekday
Ok - What exactly is happening here?
- generated links from the file temp.txt were moved to the file /var/www/html/youtube.txt
- clean up duplicates
- remove the temp file (really on /var/www/html/ and not on /etc/temp/?)
- update the pihole gravity
In which step are the links in /var/www/html/youtube.txt processed as blacklisted links?
If I audit my pihole (Tools --> Query Llsts), no single URL causes a hit to be blocked
Hi,
What is the latest link I should add to block ads on Youtube? I know the blocking is not guarantee.
Where exactly do I enter the list, in the "Adlist group management" or there is a TXT list I can edit and put many links?
Last question, it there a list for Spotify?
Waiting for the reply to it myself for PiHole v5. Will appreciate the response and guidance for it.
Thanks.
Someone...?
Hello everyone. I just registered to this website as well to help some fellow friends with the code provided by Marcio_Xavier a couple of months ago.
I've noticed that it needed some tweaks besides the ones provided by some of you already. Let me know if you find it of any use, I will post it here.
Remember that you need two files:
/etc/pihole/temp.sh --> This is the one I am providing new code below
/etc/pihole/youtube-ads.sh --> You can find this one in the original Marcio_Xavier's post. Bear in mind that you might have to delete the following line from it:
#removes the temporary file on the raspberry pi
rm /var/www/html/temp.txt
This is the new code for the /etc/pihole/temp.sh file
#!/bin/sh
# This script will fetch the Googlevideo ad domains and append them to the Pi-hole block list.
# Run this script daily with a cron job (don't forget to chmod +x)
# More info here: https://discourse.pi-hole.net/t/how-do-i-block-ads-on-youtube/253/136
# File to store the YT ad domains
FILE=/etc/temp/temp.txt
# Wolfram Alfa AppID
APPID="YOUR_APPID_HERE. You can retrieve it from your Wolfphramalpha Account"
#Verify if $FILE exists
if [ ! -e "$FILE" ] ; then
touch "$FILE"
fi
if [ ! -w "$FILE" ] ; then
echo Cannot write to $FILE
exit 1
fi
# Fetch the list of domains, remove the ip's and save them
# You are limited to use 50 queries a day. Don't use it too often
curl 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com' \
| awk -F, 'NR>1{print $1}' \
| grep -vE "redirector|manifest" >> $FILE
# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too
POD_TIMEOUT="20"
curl "http://api.wolframalpha.com/v2/query?input=googlevideo.com&appid=${APPID}&format=plaintext&podtimeout=${POD_TIMEOUT}&podstate=WebSiteStatisticsPod:Inte
rnetData__Subdomains&podstate=WebSiteStatisticsPod:InternetData__Subdomains_More" \
| grep -Po "r\d+---sn-.+.googlevideo.com" >> $FILE
# Scan log file for previously accessed domains
grep r*.googlevideo.com /var/log/pihole.log \
| awk '{print $6}' \
| grep -v '^googlevideo.com\|redirector\|manifest' \
| sort | uniq >> $FILE
# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too
cat $FILE | sed -i $FILE -re 's/(^r[[:digit:]]+)(.)/\1---/' >> $FILE
cat $FILE | sed -i $FILE -re 's/\-------/\---/' >> $FILE
cat $FILE | sed -i $FILE -re 's/\-----/\---/' >> $FILE
#delete garbage
sed -i 's/\/etc\/pihole\/lan.list//g' $FILE
sed -i 's/cached//g' $FILE
sed -i 's/forwarded//g' $FILE
sed -i 's/gravity//g' $FILE
sed -i 's/query\'['AAAA\']'//g' $FILE
sed -i 's/query\'['A\']'//g' $FILE
sed -i 's/reply//g' $FILE
sed -i 's/blocked//g' $FILE
sed -i 's/172.217.3.46//g' $FILE
sed -i '/^[[:space:]]*$/d' $FILE
#delete duplicates
perl -i -ne 'print if ! $x{$_}++' /etc/temp/temp.txt
Just a few final remarks:
-
Remember that you need to chmod +x both of these files when you're done editing them if you just created them.
-
I've tweaked my crontab -e file and I suggest you update it as well. This way you avoid the risk of exceeding your daily free quota to the HackerTarget API.
# This script will fetch the Googlevideo ad domains and append them to the Pi-hole block list
*/60 * * * * sh /etc/pihole/temp.sh
# This script will run after to update the Gravity adlists and for Youtube blocking
*/61 * * * 1-6 sh /etc/pihole/youtube-ads.sh
# You will need to check your file /etc/cron.d/pihole and look at what time Gravity updates. It should be one of the very first lines:
# Example:
# 5 3 * * 7 root PATH="$PATH:/usr/local/bin/" pihole updateGravity >/var/log/pihole_updateGravity.log || cat /var/log/pihole_updateGravity.log
# This means that on every Sunday at 3:05am gravity will update itself
# This cron will run 5 minutes prior that (3:00am) and on Sundays only
*/61 3 * * 7
- Some of you asked where you should put the blocklist that is created: http://localhost/youtube.txt . Assuming you are under the newest version of Pi-hole, you should go to your web admin interface (eg 192.168.7.103/admin) and go to Group Management -> Adlists.
Update Gravity after either inside the Web admin interface, or with pihole -g
Let me know if this helps!
I've also just registered to add my script adjustments and thoughts. I've taken what PandaWithPants and Marcio_Xavier have provided and merged it into this bash script:
#!/bin/sh
# This script will fetch the Googlevideo ad domains and append them to the Pi-hole block list.
# Run this script daily with a cron job (don't forget to chmod +x)
# More info here: https://discourse.pi-hole.net/t/how-do-i-block-ads-on-youtube/253/136
# File to store the YT ad domains
FILE=/tmp/youtube.txt
# Wolfram Alfa AppID
APPID="APPID"
#Verify if $FILE exists
if [ ! -e "$FILE" ] ; then
touch "$FILE"
fi
if [ ! -w "$FILE" ] ; then
echo Cannot write to $FILE
exit 1
fi
# Fetch the list of domains, remove the ip's and save them
# You are limited to use 50 queries a day. Don't use it too often
echo " [i] Downloading googlevideo.com hackertarget list"
curl -s 'https://api.hackertarget.com/hostsearch/?q=googlevideo.com' \
| awk -F, 'NR>1{print $1}' \
| grep -vE "redirector|manifest" >> $FILE
echo " [✓] Done"
# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too
POD_TIMEOUT="20"
echo " [i] Downloading WolframAlpha magic"
curl -s "http://api.wolframalpha.com/v2/query?input=googlevideo.com&appid=${APPID}&format=plaintext&podtimeout=${POD_TIMEOUT}&podstate=WebSiteStatisticsPod:InternetData__Subdomains&podstate=WebSiteStatisticsPod:InternetData__Subdomains_More" \
| grep -Po "r\d+---sn-.+.googlevideo.com" >> $FILE
echo " [✓] Done"
# Scan log file for previously accessed domains
grep r*.googlevideo.com /var/log/pihole.log \
| awk '{print $6}' \
| grep -v '^googlevideo.com\|redirector\|manifest' \
| sort | uniq >> $FILE
echo " [✓] Processed pihole logs"
# Replace r*.sn*.googlevideo.com URLs to r*---sn-*.googlevideo.com
# and add those to the list too
cat $FILE | sed -i $FILE -re 's/(^r[[:digit:]]+)(.)/\1---/' >> $FILE
cat $FILE | sed -i $FILE -re 's/\-------/\---/' >> $FILE
cat $FILE | sed -i $FILE -re 's/\-----/\---/' >> $FILE
echo " [✓] Added lines, because lines are cool"
#delete garbage
sed -i 's/\/etc\/pihole\/lan.list//g' $FILE
sed -i 's/cached//g' $FILE
sed -i 's/forwarded//g' $FILE
sed -i 's/gravity//g' $FILE
sed -i 's/query\'['AAAA\']'//g' $FILE
sed -i 's/query\'['A\']'//g' $FILE
sed -i 's/reply//g' $FILE
sed -i 's/blocked//g' $FILE
sed -i 's/172.217.3.46//g' $FILE
sed -i '/^[[:space:]]*$/d' $FILE
echo " [✓] Removed garbage"
#delete duplicates
perl -i -ne 'print if ! $x{$_}++' $FILE
echo " [✓] Deduplicated the temporary blocklist"
HOSTEDFILE="/var/www/html/youtube.txt"
cat $FILE >> $HOSTEDFILE
echo " [✓] Appended to the hosted blocklist"
perl -i -ne 'print if ! $x{$_}++' $HOSTEDFILE
echo " [✓] Deduplicated the hosted blacklist"
pihole -g
I've added the resulting file (http://localhost/youtube.txt) to the adlist and had this active for about 3 hours until I noticed that ads were still showing up in the Youtube Android app and some videos weren't able to play, so this sadly doesn't seem to be a viable solution.
2 Likes
I don't use Youtube too often a day, as I am on my cellphone or barely at my laptop but I have seen that some complained that "some videos won't play" by using this method.
Would you let me know which kind or which videos (send me their links pls ) so I can check as well on my end?
You are right that some videos will still play ads, some others won't. But again as I don't watch that many videos, and the ones I frequent I was able to play (both mobile and in laptop) I would like to try if the "video not loading problem" I see it too
Thanks!
1 Like
# /etc/pihole/temp.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23078 100 23078 0 0 27638 0 --:--:-- --:--:-- --:--:-- 27605
curl: (3) URL using bad/illegal format or missing URL
Thank you. The script works, but Pihole blocks too much and some videos cannot be played
I'm not 100% sure, but I believe this was one of the videos, that didn't play for me, with the Adlist active.
I had the same issue, the code block PandaWithPants posted seems to result in some issues, when directly copy/pasted. The error you noted is due to a linebreak in the line "curl "http://api.wolframalpha.com/", where the URL is split with a new line.
1 Like
Thanks for noting about the line break. Yeah copying/pasting sometimes provokes those line breaks issues, I've directly copied the code from my Cygwin terminal window.
As for the video, I was able to play it both in the app/browser. I'd suggest though again not a 100% automatic solution, that you go to your web admin interface and look at the most recent logs in the Query log. Then you'd see an r#---sn-... entry in color red in there.
You should whitelist the topmost entry of those and every time another video wouldn't load.
As far as the method is too aggresive I would like to try other methods as well but as this works for casual use for mostly everyone is what it is the best right now.
Let me know if you need any other help!
Thanks
1 Like
Thanks for the response and your insight!
Oddly enough the videos only fail to play sometimes, sometimes it works just fine. But as I said before, ads keep coming up for me with the filter active. As stated before I'm very new to Pi-Hole and never occupied myself with how Youtube delivers content/ads.
I've resorted to an alternative solution for me uBlock on the desktop works fine and I use a modded Youtube app on Android which seems to work perfectly fine without ads (I'm not sure if external links are welcome here, but I can PM anyone the link).
Yup, right now the best resort (for browsers mostly) is to use an adblocker add-on, as there is no 100% reliable solution on Pi-hole at the moment. The script blocks YT servers that serve both ads and video, so that's why sometimes a video breaks because the script blocked a "video-only server". And the other videos that are plagued with ads & video are not playable since some YT servers serve both ads & video.
For devices that are not ad-blocking capable like: ChromeCast, Smart TVs, Apple TV, Amazon Fire this works 70-85% that's why many have resorted to Pi-hole for ad-blocking to these devices.
But since now YT is aware about people trying to block these r#--- servers, they actually punish you sending more ads when you're trying to block the connections from any r#---sn-gfjdgdkfgkd... server, so this script method could be useless to many.
There is another Reddit thread that is mentioned at the very beginning of this topic, where some said to have luck by doing this:
-
nslookup manifest.googlevideo.com
-
Check what IP throws in the result: Eg: 216.58.194.206
-
nslookup 216.58.194.206
-
If you see one server in your result it could mean that is the server geographically closest to you that serves ads (in theory).
So you would then have to create a custom hosts file according to this guide and add a line like this:
216.58.194.206 manifest.googlevideo.com
Again, the IP you see could change, so you would have to do the steps in sequence.
I would like to see if there is another solution in the future, but for the most part it is what it is right now, sadly.
1 Like
Hi guys, total noob here...working my way through this. One small question on editing crontab: do I edit it for the user "pi" or as "root" for it to work?
Depends on which user you used for installation. To determine that do a cd /etc/pihole and then ls -l
You will see which user created the files in that directory. If it's pi then just do a crontab with that user logged on, otherwise switch to root with su command and then run crontab -e
1 Like
Thanks for the tip. I wonder if Google recognizes if Pi Hole blocks an ad url and replaces with a brand new one resulting in an neverending game of cat and mouse.
Also I used to run YouTube on Brave Browser and if I have block ads enabled all ads would be blocked. I'm not a web whiz so not sure if this would work: could we can find out how Brave blocks those ads and incorporate that technique into Pi Hole?