Remove dead / non existing domains from local gravity list

Feature Requests Closed or Out Of Scope
1

Hey guys, I think this might be a good idea specially where memory is becoming a constraint.

Feature:

  • Remove domains that return NXDOMAIN from the local gravity list.

Benefit:

  • Reduce the amount of memory used by the gravity list for storing dead domains.
  • Reduce heating of the device if gravity list is very large to begin with and contains a large amount of dead domains.

Possible caveat

  • There is no guarantee that the domains won't come back to life.
    • Caveat fix -> Update the domains from source once more.

Thoughts:
This option can live in the "Danger zone", it will run a script to check if the currently listed domains in the local blocklist returns NXDOMAIN and if it does, remove said domain from the list ( freeing up some mem space ).

I have a pi 3b+ and have only added one additional third party block-list and that has increased my total blocked domains to roughly 1.5M putting my memory usage around 20-23%. Now for the 3b+ that's not an issue since it has 1GB of ram, but if you are running / wanting to run Pi-hole on an earlier Pi release then it would become problematic.

I think this feature is pretty safe, the only worry I might have is how exactly the pinging / checking of the domains existence might work and if the users could exploit it.
Awesome work so far, keep it up :wink:

2

Memory isn't the same constraint in version 5 as was previously.

3

So in that case is there no real benefit to reducing the size of the gravity list? (V5+)

4

The domains are no longer loaded in to memory, just the pointers to where the domains are stored in the database.

I can't speak to benefits or not with reducing the number of blocked domains.

5

Why should this be something that every Pi-Hole install does? This should be done at the block list maintainer level. They would do this one time, then send out a clean list of domains. Then, tens of thousands of Pi-Hole installs won't have to do the same thing.

6

A valid point, I did think about that while suggesting this feature request.

Yes the maintainer should do this and I'm sure the default ones that Pi-hole suggests do, but there are many third party ones that are more robust and less... cared for, so the likelihood of those containing dead domains I feel is quite high.

So basically due to the lack of trust on the maintenance of those blocklists I thought having the power to clean up my gravity list to its minimal size without losing anything may improve the current problem of running out of memory V4.x. ( Which apparently is handled in V5+)

If it tells you which lists has the dead domains it could also help you identify the lists that are actually cared for which may fall under the blocklist management umbrella.

Whole thing becomes moot if there is no real performance or significant memory enhancement though
¯_(ツ)_/¯

7

You shouldn't use lists you don't trust.

1 Like