Limit to gravity.db size?

Maverick · February 20, 2020, 9:31am

Please follow the below template, it will help us to help you!

Please ensure that you are running the latest version of the beta code.
Run pihole -up to update to the latest, then verify that the problem still exists before reporting it.

Problem with Beta 5.0:
I updated a 4.3.2 version to 5 beta and I noticed that the domains trapped in the event horizon changed drastically.
*** Prior****
[✓] Consolidating blocklists
[✓] Extracting domains from blocklists
[i] Number of domains being pulled in by gravity: 18964480
[✓] Removing duplicate domains
[i] Number of unique domains trapped in the Event Horizon: 13073583
[i] Number of whitelisted domains: 1409

*** After ***
[✓] Storing downloaded domains in new gravity database
[✓] Building tree
[✓] Swapping databases
[i] Number of gravity domains: 6930460 (2911027 unique domains)
[i] Number of exact blacklisted domains: 0
[i] Number of regex blacklist filters: 0
[i] Number of exact whitelisted domains: 1409

The same sources are being downloaded.

I'm running a updated docker image. I've tried this twice, once as a fresh build and once as an upgrade using the previous data folder with the same results. Any ideas?

Debug Token:
https://tricorder.pi-hole.net/j9ytqmm3qz

DL6ER · February 20, 2020, 7:39pm

There is no limit to the gravity database size. From your debug log, I see you're using 146 adlists. Have you checked them all for suitable format?

Please try

pihole checkout core fix/gravity_add_newline

This also shows non-critical warnings which may result in a reduction of imported domains during a pihole -g run.

Maverick · February 20, 2020, 8:15pm

Executed

pihole checkout core fix/gravity_add_newline

Current Version Now

  Current Pi-hole version is v4.3.2-170-g24241f6
  Current AdminLTE version is v4.3.2-375-gdd7b2b20
  Current FTL version is vDev-61d67ff

Previous Version

  Pi-hole version is v4.3.2-181-gddbd57f (Latest: v4.3.2)
  AdminLTE version is v4.3.2-370-g4e27eba0 (Latest: v4.3.3)
  FTL version is vDev-ff840d0 (Latest: v4.3.1)

Version I am testing against.

  Pi-hole version is v4.3.2 (Latest: v4.3.2)
  AdminLTE version is v4.3.2 (Latest: v4.3.3)
  FTL version is v4.3.1 (Latest: v4.3.1)

  Encountered non-critical SQL warnings. Please check the suitability of the list you're using!
SQL warnings:
/tmp/tmp.aEUCza3awc.gravity:1231532: expected 2 columns but found 3 - extras ignored

Was the only thing found along with 3 sources that failed to download. I'm removing the three sources from both installs and performing pihole -g again, which results in a difference of less than 100K currently.

Latest pihole -g
Results: 4.3.2 Release

  [✓] Consolidating blocklists
  [✓] Extracting domains from blocklists
  [i] Number of domains being pulled in by gravity: 17828604
  [✓] Removing duplicate domains
  [i] Number of unique domains trapped in the Event Horizon: 12974491
  [i] Number of whitelisted domains: 1372
  [i] Number of regex filters: 0
  [✓] Parsing domains into hosts format
  [✓] Cleaning up stray matter

Results: 4.3.2-170-g24241f6

  [✓] Storing downloaded domains in new gravity database
  Encountered non-critical SQL warnings. Please check the suitability of the list you're using!
SQL warnings:
/tmp/tmp.vHt9HaOUDV.gravity:1031058: expected 2 columns but found 3 - extras ignored

  [✓] Building tree
  [✓] Swapping databases
  [✓] Flushing DNS cache
  [i] Number of gravity domains: 7075635 (2912064 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 1411
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

As far as file format. I do have a single file myself that is just under 1GB that I created from a SQL query that is correctly formatted. It currently contains a little over 11million, which is more than what is displayed here as a whole.

Maverick · February 20, 2020, 8:22pm

I just updated to

  Pi-hole version is v4.3.2-190-g85c15a7 (Latest: v4.3.2)
  AdminLTE version is v4.3.2-375-gdd7b2b20 (Latest: v4.3.3)
  FTL version is vDev-61d67ff (Latest: v4.3.1)

and the SQL error is no longer present, but the host numbers are about the same.

  [✓] Storing downloaded domains in new gravity database
  [✓] Building tree
  [✓] Swapping databases
  [i] Number of gravity domains: 7075585 (2912057 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 1411
  [i] Number of regex whitelist filters: 0
  [✓] Cleaning up stray matter

DL6ER · February 20, 2020, 8:27pm

I cannot really reproduce this as my Pi-hole hardware is not even near of being able to handle this number of domains in any acceptable time. Could you count the number of lines in the lists you're trying to download?

They're stored in /etc/pihole/lists.* and

cat /etc/pihole/list.* | wc -l

should give you a rough estimate.

Please run also

grep -c ^ /etc/pihole/list.*

Maverick · February 20, 2020, 8:40pm

root@pihole:/# cat /etc/pihole/list.* | wc -l
7066281

I just noticed something
I'll just include one for an example.

grep -c ^ /etc/pihole/list.*
/etc/pihole/list.128.10.0.0.1.domains:0

versus

root@pihole-pub:/etc/pihole# cat /etc/pihole/list.* | wc -l
18409032

*Edited to remove list.preEventHorizon

grep -c ^ /etc/pihole/list.*
/etc/pihole/list.128.10.0.0.1.domains:11878956

Maverick · February 20, 2020, 8:45pm

root@pihole:/# grep -c ^ /etc/pihole/list.*
Dev

/etc/pihole/list.0.raw.githubusercontent.com.domains:51283
/etc/pihole/list.1.mirror1.malwaredomains.com.domains:26860
/etc/pihole/list.10.raw.githubusercontent.com.domains:2701
/etc/pihole/list.100.someonewhocares.org.domains:14328
/etc/pihole/list.101.raw.githubusercontent.com.domains:7066
/etc/pihole/list.102.raw.githubusercontent.com.domains:8881
/etc/pihole/list.103.winhelp2002.mvps.org.domains:10190
/etc/pihole/list.104.hostsfile.mine.nu.domains:105863
/etc/pihole/list.105.v.firebog.net.domains:7642
/etc/pihole/list.106.adblock.mahakala.is.domains:255375
/etc/pihole/list.107.dbl.oisd.nl.domains:1001329
/etc/pihole/list.108.raw.githubusercontent.com.domains:5
/etc/pihole/list.109.raw.githubusercontent.com.domains:3
/etc/pihole/list.11.raw.githubusercontent.com.domains:2735
/etc/pihole/list.110.jasonhill.co.uk.domains:1754
/etc/pihole/list.111.raw.githubusercontent.com.domains:418
/etc/pihole/list.112.raw.githubusercontent.com.domains:3263
/etc/pihole/list.113.phishing.mailscanner.info.domains:4396
/etc/pihole/list.114.zerodot1.gitlab.io.domains:46752
/etc/pihole/list.115.zerodot1.gitlab.io.domains:3304
/etc/pihole/list.116.zerodot1.gitlab.io.domains:775
/etc/pihole/list.117.gist.githubusercontent.com.domains:16276
/etc/pihole/list.118.gist.githubusercontent.com.domains:1110
/etc/pihole/list.119.raw.githubusercontent.com.domains:881
/etc/pihole/list.12.raw.githubusercontent.com.domains:1
/etc/pihole/list.120.raw.githubusercontent.com.domains:976
/etc/pihole/list.121.raw.githubusercontent.com.domains:305
/etc/pihole/list.122.raw.githubusercontent.com.domains:959551
/etc/pihole/list.123.raw.githubusercontent.com.domains:9330
/etc/pihole/list.124.github.com.domains:1325391
/etc/pihole/list.125.github.com.domains:364
/etc/pihole/list.126.raw.githubusercontent.com.domains:4657
/etc/pihole/list.127.v.firebog.net.domains:29835
/etc/pihole/list.128.10.0.0.1.domains:0
/etc/pihole/list.129.blocklist.site.domains:148028
/etc/pihole/list.13.raw.githubusercontent.com.domains:34
/etc/pihole/list.130.blocklist.site.domains:196031
/etc/pihole/list.131.blocklist.site.domains:26590
/etc/pihole/list.132.blocklist.site.domains:148028
/etc/pihole/list.133.blocklist.site.domains:196031
/etc/pihole/list.134.blocklist.site.domains:26590
/etc/pihole/list.135.blocklist.site.domains:667
/etc/pihole/list.136.blocklist.site.domains:26590
/etc/pihole/list.137.blocklist.site.domains:719243
/etc/pihole/list.138.blocklist.site.domains:189974
/etc/pihole/list.139.blocklist.site.domains:7424
/etc/pihole/list.14.raw.githubusercontent.com.domains:1109
/etc/pihole/list.140.blocklist.site.domains:1903
/etc/pihole/list.141.blocklist.site.domains:108628
/etc/pihole/list.142.blocklist.site.domains:227
/etc/pihole/list.15.raw.githubusercontent.com.domains:2597
/etc/pihole/list.16.raw.githubusercontent.com.domains:1058
/etc/pihole/list.17.raw.githubusercontent.com.domains:1286
/etc/pihole/list.18.raw.githubusercontent.com.domains:20
/etc/pihole/list.19.raw.githubusercontent.com.domains:2578
/etc/pihole/list.2.sysctl.org.domains:20567
/etc/pihole/list.20.raw.githubusercontent.com.domains:73
/etc/pihole/list.21.raw.githubusercontent.com.domains:7458
/etc/pihole/list.22.raw.githubusercontent.com.domains:1104
/etc/pihole/list.23.raw.githubusercontent.com.domains:3203
/etc/pihole/list.24.raw.githubusercontent.com.domains:26858
/etc/pihole/list.25.raw.githubusercontent.com.domains:1738
/etc/pihole/list.26.raw.githubusercontent.com.domains:879
/etc/pihole/list.27.raw.githubusercontent.com.domains:3301
/etc/pihole/list.29.raw.githubusercontent.com.domains:14393
/etc/pihole/list.3.zeustracker.abuse.ch.domains:0
/etc/pihole/list.30.raw.githubusercontent.com.domains:7064
/etc/pihole/list.31.raw.githubusercontent.com.domains:1170
/etc/pihole/list.32.raw.githubusercontent.com.domains:10652
/etc/pihole/list.34.raw.githubusercontent.com.domains:371
/etc/pihole/list.35.raw.githubusercontent.com.domains:14335
/etc/pihole/list.36.raw.githubusercontent.com.domains:807
/etc/pihole/list.37.www.stopforumspam.com.domains:11131
/etc/pihole/list.38.raw.githubusercontent.com.domains:57019
/etc/pihole/list.39.justdomains.github.io.domains:16022
/etc/pihole/list.4.s3.amazonaws.com.domains:34
/etc/pihole/list.40.justdomains.github.io.domains:16022
/etc/pihole/list.41.justdomains.github.io.domains:6581
/etc/pihole/list.42.github.com.domains:2
/etc/pihole/list.43.github.com.domains:2
/etc/pihole/list.44.github.com.domains:2
/etc/pihole/list.45.github.com.domains:2
/etc/pihole/list.46.github.com.domains:2
/etc/pihole/list.47.github.com.domains:2
/etc/pihole/list.48.github.com.domains:2
/etc/pihole/list.49.github.com.domains:2
/etc/pihole/list.5.s3.amazonaws.com.domains:2701
/etc/pihole/list.50.github.com.domains:2
/etc/pihole/list.51.github.com.domains:2
/etc/pihole/list.52.github.com.domains:2
/etc/pihole/list.53.raw.githubusercontent.com.domains:692
/etc/pihole/list.54.jasonhill.co.uk.domains:1754
/etc/pihole/list.55.adaway.org.domains:12166
/etc/pihole/list.56.v.firebog.net.domains:28436
/etc/pihole/list.57.raw.githubusercontent.com.domains:38353
/etc/pihole/list.58.v.firebog.net.domains:1578
/etc/pihole/list.59.pgl.yoyo.org.domains:3306
/etc/pihole/list.6.hosts-file.net.domains:45736
/etc/pihole/list.60.raw.githubusercontent.com.domains:10
/etc/pihole/list.61.v.firebog.net.domains:2611
/etc/pihole/list.62.v.firebog.net.domains:3671
/etc/pihole/list.63.gitlab.com.domains:832
/etc/pihole/list.64.raw.githubusercontent.com.domains:1286
/etc/pihole/list.65.raw.githubusercontent.com.domains:365
/etc/pihole/list.66.raw.githubusercontent.com.domains:83
/etc/pihole/list.67.raw.githubusercontent.com.domains:185
/etc/pihole/list.68.v.firebog.net.domains:50238
/etc/pihole/list.69.s3.amazonaws.com.domains:2735
/etc/pihole/list.7.raw.githubusercontent.com.domains:12165
/etc/pihole/list.70.hosts-file.net.domains:1156
/etc/pihole/list.71.hosts-file.net.domains:300282
/etc/pihole/list.72.hosts-file.net.domains:1156
/etc/pihole/list.73.mirror.cedia.org.ec.domains:3196
/etc/pihole/list.74.www.malwaredomainlist.com.domains:1104
/etc/pihole/list.75.bitbucket.org.domains:2046
/etc/pihole/list.76.v.firebog.net.domains:29835
/etc/pihole/list.77.phishing.army.domains:13857
/etc/pihole/list.78.gitlab.com.domains:0
/etc/pihole/list.79.ransomwaretracker.abuse.ch.domains:0
/etc/pihole/list.8.raw.githubusercontent.com.domains:28540
/etc/pihole/list.80.ransomwaretracker.abuse.ch.domains:0
/etc/pihole/list.81.ransomwaretracker.abuse.ch.domains:0
/etc/pihole/list.82.ransomwaretracker.abuse.ch.domains:0
/etc/pihole/list.83.ransomwaretracker.abuse.ch.domains:0
/etc/pihole/list.84.v.firebog.net.domains:19586
/etc/pihole/list.85.raw.githubusercontent.com.domains:2578
/etc/pihole/list.86.raw.githubusercontent.com.domains:221319
/etc/pihole/list.87.v.firebog.net.domains:47399
/etc/pihole/list.88.hosts-file.net.domains:524
/etc/pihole/list.89.reddestdream.github.io.domains:710
/etc/pihole/list.9.raw.githubusercontent.com.domains:38251
/etc/pihole/list.90.raw.githubusercontent.com.domains:7416
/etc/pihole/list.91.raw.githubusercontent.com.domains:73
/etc/pihole/list.92.v.firebog.net.domains:720
/etc/pihole/list.93.v.firebog.net.domains:8963
/etc/pihole/list.94.www.dshield.org.domains:98459
/etc/pihole/list.95.www.dshield.org.domains:2573
/etc/pihole/list.96.www.dshield.org.domains:2286
/etc/pihole/list.97.www.joewein.net.domains:69333
/etc/pihole/list.98.raw.githubusercontent.com.domains:1738
/etc/pihole/list.99.hostsfile.org.domains:25263

Release:

/etc/pihole/list.0.raw.githubusercontent.com.domains:60521
/etc/pihole/list.1.mirror1.malwaredomains.com.domains:26861
/etc/pihole/list.10.raw.githubusercontent.com.domains:2701
/etc/pihole/list.100.someonewhocares.org.domains:14728
/etc/pihole/list.101.raw.githubusercontent.com.domains:7066
/etc/pihole/list.102.raw.githubusercontent.com.domains:8889
/etc/pihole/list.103.winhelp2002.mvps.org.domains:12033
/etc/pihole/list.104.hostsfile.mine.nu.domains:106362
/etc/pihole/list.105.v.firebog.net.domains:7642
/etc/pihole/list.106.adblock.mahakala.is.domains:255392
/etc/pihole/list.108.raw.githubusercontent.com.domains:5
/etc/pihole/list.109.raw.githubusercontent.com.domains:3
/etc/pihole/list.11.raw.githubusercontent.com.domains:2735
/etc/pihole/list.110.jasonhill.co.uk.domains:1782
/etc/pihole/list.111.raw.githubusercontent.com.domains:422
/etc/pihole/list.112.raw.githubusercontent.com.domains:3263
/etc/pihole/list.113.phishing.mailscanner.info.domains:4401
/etc/pihole/list.114.zerodot1.gitlab.io.domains:46752
/etc/pihole/list.115.zerodot1.gitlab.io.domains:3304
/etc/pihole/list.116.zerodot1.gitlab.io.domains:775
/etc/pihole/list.117.gist.githubusercontent.com.domains:16544
/etc/pihole/list.118.gist.githubusercontent.com.domains:1113
/etc/pihole/list.119.raw.githubusercontent.com.domains:931
/etc/pihole/list.12.raw.githubusercontent.com.domains:1
/etc/pihole/list.120.raw.githubusercontent.com.domains:982
/etc/pihole/list.121.raw.githubusercontent.com.domains:311
/etc/pihole/list.122.raw.githubusercontent.com.domains:959698
/etc/pihole/list.123.raw.githubusercontent.com.domains:9330
/etc/pihole/list.124.github.com.domains:1325539
/etc/pihole/list.125.github.com.domains:370
/etc/pihole/list.126.raw.githubusercontent.com.domains:4659
/etc/pihole/list.127.v.firebog.net.domains:29835
/etc/pihole/list.128.10.0.0.1.domains:11878956
/etc/pihole/list.129.blocklist.site.domains:148040
/etc/pihole/list.13.raw.githubusercontent.com.domains:34
/etc/pihole/list.130.blocklist.site.domains:196032
/etc/pihole/list.131.blocklist.site.domains:26591
/etc/pihole/list.132.blocklist.site.domains:148040
/etc/pihole/list.133.blocklist.site.domains:196032
/etc/pihole/list.134.blocklist.site.domains:26591
/etc/pihole/list.135.blocklist.site.domains:668
/etc/pihole/list.136.blocklist.site.domains:26591
/etc/pihole/list.137.blocklist.site.domains:719522
/etc/pihole/list.138.blocklist.site.domains:189977
/etc/pihole/list.139.blocklist.site.domains:7436
/etc/pihole/list.14.raw.githubusercontent.com.domains:1109
/etc/pihole/list.140.blocklist.site.domains:1905
/etc/pihole/list.141.blocklist.site.domains:108630
/etc/pihole/list.142.blocklist.site.domains:229
/etc/pihole/list.15.raw.githubusercontent.com.domains:2597
/etc/pihole/list.16.raw.githubusercontent.com.domains:1058
/etc/pihole/list.17.raw.githubusercontent.com.domains:1286
/etc/pihole/list.18.raw.githubusercontent.com.domains:20
/etc/pihole/list.19.raw.githubusercontent.com.domains:2578
/etc/pihole/list.2.sysctl.org.domains:20569
/etc/pihole/list.20.raw.githubusercontent.com.domains:73
/etc/pihole/list.21.raw.githubusercontent.com.domains:7458
/etc/pihole/list.22.raw.githubusercontent.com.domains:1104
/etc/pihole/list.23.raw.githubusercontent.com.domains:3203
/etc/pihole/list.24.raw.githubusercontent.com.domains:26858
/etc/pihole/list.25.raw.githubusercontent.com.domains:1738
/etc/pihole/list.26.raw.githubusercontent.com.domains:879
/etc/pihole/list.27.raw.githubusercontent.com.domains:3301
/etc/pihole/list.29.raw.githubusercontent.com.domains:14393
/etc/pihole/list.3.zeustracker.abuse.ch.domains:1
/etc/pihole/list.30.raw.githubusercontent.com.domains:7064
/etc/pihole/list.31.raw.githubusercontent.com.domains:1170
/etc/pihole/list.32.raw.githubusercontent.com.domains:10652
/etc/pihole/list.34.raw.githubusercontent.com.domains:371
/etc/pihole/list.35.raw.githubusercontent.com.domains:14335
/etc/pihole/list.36.raw.githubusercontent.com.domains:807
/etc/pihole/list.37.www.stopforumspam.com.domains:11131
/etc/pihole/list.38.raw.githubusercontent.com.domains:57046
/etc/pihole/list.39.justdomains.github.io.domains:16022
/etc/pihole/list.4.s3.amazonaws.com.domains:2707
/etc/pihole/list.40.justdomains.github.io.domains:6581
/etc/pihole/list.41.justdomains.github.io.domains:28522
/etc/pihole/list.42.github.com.domains:1270
/etc/pihole/list.43.github.com.domains:4214
/etc/pihole/list.44.github.com.domains:8077
/etc/pihole/list.45.github.com.domains:7149
/etc/pihole/list.46.github.com.domains:3417
/etc/pihole/list.47.github.com.domains:58397
/etc/pihole/list.48.github.com.domains:108065
/etc/pihole/list.49.github.com.domains:906
/etc/pihole/list.5.s3.amazonaws.com.domains:2705
/etc/pihole/list.50.github.com.domains:1113
/etc/pihole/list.51.github.com.domains:33687
/etc/pihole/list.52.github.com.domains:889
/etc/pihole/list.53.raw.githubusercontent.com.domains:704
/etc/pihole/list.54.jasonhill.co.uk.domains:1782
/etc/pihole/list.55.adaway.org.domains:18995
/etc/pihole/list.56.v.firebog.net.domains:28436
/etc/pihole/list.57.raw.githubusercontent.com.domains:38362
/etc/pihole/list.58.v.firebog.net.domains:1578
/etc/pihole/list.59.pgl.yoyo.org.domains:3320
/etc/pihole/list.6.hosts-file.net.domains:45747
/etc/pihole/list.60.raw.githubusercontent.com.domains:10
/etc/pihole/list.61.v.firebog.net.domains:2611
/etc/pihole/list.62.v.firebog.net.domains:3671
/etc/pihole/list.63.gitlab.com.domains:14771
/etc/pihole/list.64.raw.githubusercontent.com.domains:1286
/etc/pihole/list.65.raw.githubusercontent.com.domains:371
/etc/pihole/list.66.raw.githubusercontent.com.domains:111
/etc/pihole/list.67.raw.githubusercontent.com.domains:255
/etc/pihole/list.68.v.firebog.net.domains:50239
/etc/pihole/list.69.s3.amazonaws.com.domains:2739
/etc/pihole/list.7.raw.githubusercontent.com.domains:12165
/etc/pihole/list.70.hosts-file.net.domains:1166
/etc/pihole/list.71.hosts-file.net.domains:300292
/etc/pihole/list.72.hosts-file.net.domains:189148
/etc/pihole/list.73.mirror.cedia.org.ec.domains:3197
/etc/pihole/list.74.www.malwaredomainlist.com.domains:1110
/etc/pihole/list.75.bitbucket.org.domains:2046
/etc/pihole/list.76.v.firebog.net.domains:29835
/etc/pihole/list.77.phishing.army.domains:13869
/etc/pihole/list.78.gitlab.com.domains:217
/etc/pihole/list.79.ransomwaretracker.abuse.ch.domains:1
/etc/pihole/list.8.raw.githubusercontent.com.domains:28540
/etc/pihole/list.80.ransomwaretracker.abuse.ch.domains:1
/etc/pihole/list.81.ransomwaretracker.abuse.ch.domains:1
/etc/pihole/list.82.ransomwaretracker.abuse.ch.domains:1
/etc/pihole/list.83.ransomwaretracker.abuse.ch.domains:1
/etc/pihole/list.84.v.firebog.net.domains:19587
/etc/pihole/list.85.raw.githubusercontent.com.domains:2578
/etc/pihole/list.86.raw.githubusercontent.com.domains:221345
/etc/pihole/list.87.v.firebog.net.domains:47400
/etc/pihole/list.88.hosts-file.net.domains:534
/etc/pihole/list.89.reddestdream.github.io.domains:715
/etc/pihole/list.9.raw.githubusercontent.com.domains:38251
/etc/pihole/list.90.raw.githubusercontent.com.domains:7448
/etc/pihole/list.91.raw.githubusercontent.com.domains:73
/etc/pihole/list.92.v.firebog.net.domains:787
/etc/pihole/list.93.v.firebog.net.domains:8963
/etc/pihole/list.94.www.dshield.org.domains:98487
/etc/pihole/list.95.www.dshield.org.domains:2601
/etc/pihole/list.96.www.dshield.org.domains:2314
/etc/pihole/list.97.www.joewein.net.domains:69333
/etc/pihole/list.98.raw.githubusercontent.com.domains:1738
/etc/pihole/list.99.hostsfile.org.domains:25586

DL6ER · February 20, 2020, 9:11pm

Okay, so something seems to have changed in the digestion of domains. I know we've looked at this code a lot and tuned it for performance. Pinging @PromoFaux and @DanSchaper.

Maverick · February 20, 2020, 9:18pm

Awesome! Glad this was discovered, people with smaller lists may have not noticed it as easily due to the casual fluctuation within lists. I have a SQL database full of almost 60 million of IPs and almost 11.8 million host names. I am looking to minimize this list down to one or two and update from the database. I just haven't got everything in there yet.

DanSchaper · February 20, 2020, 10:23pm

Can you get a diff on a smaller list to see what is missing/added between the versions?

Maverick · February 20, 2020, 10:27pm

I think that would be possible, give me a moment. I'll edit this post.

DanSchaper · February 20, 2020, 10:44pm

Or you can use the pihole tricorder command to get it to us privately, if needed.

Maverick · February 20, 2020, 11:22pm

Seems my numbers are off between the lists on a couple. I must have not removed one that I needed to along the way.

Anyways, seems what it is skipping is the HOSTS format files. I can't confirm this yet, still looking.

DL6ER · February 24, 2020, 9:43am

We have merged some improvements to gravity, please update and check again with pihole -g whether this provides more insight for you now.

DL6ER · April 13, 2020, 8:05am

6 posts were split to a new topic: Hosts file comments