Updating Ad Lists Fails with 1.9m+ domains

Help
1

Please follow the below template, it will help us to help you!

Most of the time, when I run pihole -g from CLi or Update Lists from Web UI I end up with an empty gravity.list and nothing being blocked. One of my block lists is particularly large, 1.9m records large. (It's https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list in case anyone wants to try reproducing).
I say most because it does sometimes work. 1/5 of my last tries worked. If I disable the big block list, it works just fine every time.

Expected Behaviour:

Lists are consolidated into /etc/pihole/gravity.list and domains blocked.

Actual Behaviour:

pihole -g ends unexpectedly and I get booted from my SSH session. Reconnecting, my /etc/pihole/gravity.list is empty.

I tried running in screen and using tee to capture output. pihole -g | tee gravity.log and got this.

  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [i] Target: raw.githubusercontent.com (hosts)
  [✓] Status: Retrieval successful

  [i] Target: mirror1.malwaredomains.com (justdomains)
  [✓] Status: No changes detected

  [i] Target: sysctl.org (hosts)
  [✓] Status: No changes detected

  [i] Target: zeustracker.abuse.ch (blocklist.php?download=domainblocklist)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_tracking.txt)
  [✓] Status: No changes detected

  [i] Target: s3.amazonaws.com (simple_ad.txt)
  [✓] Status: No changes detected

  [i] Target: hosts-file.net (ad_servers.txt)
  [✓] Status: No changes detected

  [i] Target: raw.githubusercontent.com (pi_blocklist_porn_all.list)
  [✓] Status: Retrieval successful

  [i] Target: raw.githubusercontent.com (block.list)
  [✓] Status: Retrieval successful

  [✓] Consolidating blocklists
  [i] Extracting domains from blocklists...

It ends there and doesn't finish.

2

try sudo bash -vx /opt/pihole/gravity.sh and pastebin the result this will output a LOT so be prepared for it also it will help you troubleshoot

3

Yes, it would be helpful to see this output and you can send it to us using tricorder.

cat /some/file.output | pihole tricorder
4

sudo bash -vx /opt/pihole/gravity.sh > /home/pi/loggedgravity.txt && cat /home/pi/loggedgravity.txt | pihole tricorder

should be a one liner to do it. after testing if it doesnt work to upload retry the command after the && sometimes resolution right after gravity is run take an extra moment

5

Thanks for the guidance. I had to use the following since most of the debug is spit out on stderr
sudo bash -vx /opt/pihole/gravity.sh 2>&1 | tee /home/pi/loggedgravity.txt
cat /home/pi/loggedgravity.txt | pihole tricorder
My SSH session got dropped when running the gravity.sh (like I mentioned in the OP) so the output ends where that happened.
Here's the debug code I got from tricorder.
g49myqlioj

1 Like
6

Not much more in terms of clues...

e[K  [✓] Consolidating blocklists
+ gravity_SortAndFilterConsolidatedList
+ local str num
+ str='Extracting domains from blocklists'
+ echo -ne '  [i] Extracting domains from blocklists...'
  [i] Extracting domains from blocklists...+ gravity_ParseFileIntoDomains /etc/pihole/pihole.0.matterandlight.txt /etc/pihole/pihole.1.parsedmatter.txt
+ local source=/etc/pihole/pihole.0.matterandlight.txt destination=/etc/pihole/pihole.1.parsedmatter.txt firstLine abpFilter
+ [[ /etc/pihole/pihole.0.matterandlight.txt == \/\e\t\c\/\p\i\h\o\l\e\/\p\i\h\o\l\e\.\0\.\m\a\t\t\e\r\a\n\d\l\i\g\h\t\.\t\x\t ]]
+ awk -F '#' '{print $1}'
+ awk -F / '{print $1}'
+ sed -nr -e 's/\.{2,}/./g' -e '/\./p'
+ awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}'

Can you run the command locally instead of over SSH?

7

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.