Support for compressed (as in gzip) lists


it would be quite nice to have support for compressed (as in zip, gzip, bz2, ...) filterlists.As these are pure text they are easy to compress.

Example (

root@lxc-pihole:~/test# ls -lh
total 11M
-rw-r--r-- 1 root root 11M Aug 7 09:08 ultimate.txt
root@lxc-pihole:~/test# time gzip ultimate.txt

root@lxc-pihole:~/test# ls -lh
total 3.1M
-rw-r--r-- 1 root root 3.1M Aug 7 09:08 ultimate.txt.gz

gzip without setting a compression rate reduced the size of the list from 12 to 3.1 MByte.

One can store bigger filter lists on Github
Less storage on Github
Faster Downloads

Decompression will take time, but most likely less time than saved by now effectivly faster download, assuming useful compression algos

We already download the lists compressed if the server and the client support it

Yes, but this is not quite the same as my feature request. Compressed files are smaller, and people could host larger lists without splitting them and better compression than libz is available (and it wouldn't be much of a problem to support a wide variety of compression algorithms (lzma, gzip, bz2, rar, zip), with lzma being preferred for plain text.

Even using lzma and -e(extreme):

root@lxc-pihole:~/test# time lzma -e -z ultimate.txt


root@lxc-pihole:~/test# ls -lh
total 2.6M
-rw-r--r-- 1 root root 2.6M Aug 7 09:08 ultimate.txt.lzma

root@lxc-pihole:~/test# time lzma -d ultimate.txt.lzma


root@lxc-pihole:~/test# ls -lh
total 11M
-rw-r--r-- 1 root root 11M Aug 7 09:08 ultimate.txt

Yes, compressing this takes time, decompression time however is not an issue - and the ratio being 11M to 2.6M - which could solve a lot of issues esp. for people hosting stuff on github.

I guess you'd primarily have to evaluate this with list maintainers.

If you'd store and serve compressed files, you would force any user (producers and consumer) to download the compressed archive completely before being able to inspect or amend its contents, and you'd potentially deprive any server-side compression measures of being applied effectively.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.