Support for compressed (as in gzip) lists

Feature Requests
1

Hi,

it would be quite nice to have support for compressed (as in zip, gzip, bz2, ...) filterlists.As these are pure text they are easy to compress.

Example (https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/ultimate.txt):

root@lxc-pihole:~/test# ls -lh
total 11M
-rw-r--r-- 1 root root 11M Aug 7 09:08 ultimate.txt
root@lxc-pihole:~/test# time gzip ultimate.txt

real0m0.943s
user0m0.923s
sys0m0.016s
root@lxc-pihole:~/test# ls -lh
total 3.1M
-rw-r--r-- 1 root root 3.1M Aug 7 09:08 ultimate.txt.gz
root@lxc-pihole:~/test#

gzip without setting a compression rate reduced the size of the list from 12 to 3.1 MByte.

Pro:
One can store bigger filter lists on Github
Less storage on Github
Faster Downloads

Cons:
Decompression will take time, but most likely less time than saved by now effectivly faster download, assuming useful compression algos

2

We already download the lists compressed if the server and the client support it

3

Yes, but this is not quite the same as my feature request. Compressed files are smaller, and people could host larger lists without splitting them and better compression than libz is available (and it wouldn't be much of a problem to support a wide variety of compression algorithms (lzma, gzip, bz2, rar, zip), with lzma being preferred for plain text.

4

Even using lzma and -e(extreme):

root@lxc-pihole:~/test# time lzma -e -z ultimate.txt

real0m10.338s
user0m10.134s
sys0m0.188s

root@lxc-pihole:~/test# ls -lh
total 2.6M
-rw-r--r-- 1 root root 2.6M Aug 7 09:08 ultimate.txt.lzma

root@lxc-pihole:~/test# time lzma -d ultimate.txt.lzma

real0m0.442s
user0m0.419s
sys0m0.021s

root@lxc-pihole:~/test# ls -lh
total 11M
-rw-r--r-- 1 root root 11M Aug 7 09:08 ultimate.txt

Yes, compressing this takes time, decompression time however is not an issue - and the ratio being 11M to 2.6M - which could solve a lot of issues esp. for people hosting stuff on github.

5

I guess you'd primarily have to evaluate this with list maintainers.

If you'd store and serve compressed files, you would force any user (producers and consumer) to download the compressed archive completely before being able to inspect or amend its contents, and you'd potentially deprive any server-side compression measures of being applied effectively.

6

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.