Journaling
iotop revealed the service btrfs-transacti writing the most. I chose btrfs over ext4 over both VMs for the sake of testing. (I should have known "if it ain't broke, don't fix it") This could be mitigated by adding "nodatacow" to fstab. I installed my VM on ext2 to rule out any journaling overhead that ext4 or btrfs might have.
iostat/sysstat
One thing to note is that, pihole's write stat is always zero in iotop. Besides that, in all of the VMs I made, pihole's write_bytes is more than double that of dnsmasq. Furthermore, writes to /sda could be explained only by adding those two, because if it wasn't for pihole write stats, over 50% of writes to /sda is unaccounted for.
iostat -m
Linux 4.9.0-3-amd64 (pid) 07/06/2017 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.40 0.00 0.15 0.02 0.00 99.43
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
sda 0.43 0.00 0.00 136 159
cat /proc/$(pidof dnsmasq)/io
rchar: 1040236
wchar: 6227861
syscr: 268
syscw: 85777
read_bytes: 1056768
write_bytes: 11218944
cancelled_write_bytes: 0
cat /proc/$(pidof pihole-FTL)/io
rchar: 75082562
wchar: 42972015
syscr: 58405
syscw: 23558
read_bytes: 4812800
write_bytes: 54169600
cancelled_write_bytes: 0
More finding
Additionally, I did a systemtap trace to get some realistic I/O numbers and it tells a similar story. I printed out the filesizes after the trace was stopped, and dnsmasq's io & filesize are accurate.
643 sshd r: 653 KiB w: 1 MiB
584 DB r: 462 KiB w: 979 KiB
584 loganalyzer r: 1 MiB w: 0 B
657 stapio r: 603 KiB w: 627 KiB
692 sshd r: 798 KiB w: 29 KiB
251 vmtoolsd r: 591 KiB w: 0 B
556 dnsmasq r: 0 B w: 259 KiB
594 lighttpd r: 193 KiB w: 32 KiB
712 sshd r: 130 KiB w: 46 KiB
703 cron r: 59 KiB w: 1 B
^Cls -lh /etc/pihole/pihole-FTL.db /var/log/pihole-FTL.log /var/log/pihole.logroot@debian7:~# ls -lh /etc/pihole/pihole-FTL.db /var/log/pihole-FTL.log /var/log/pihole.log
-rw-r--r-- 1 pihole pihole 60K Jul 6 04:37 /etc/pihole/pihole-FTL.db
-rw-r--r-- 1 pihole pihole 4.6K Jul 6 04:17 /var/log/pihole-FTL.log
-rw-r--r-- 1 dnsmasq root 276K Jul 6 04:37 /var/log/pihole.log
traceio cumulative systemtap script https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/SystemTap_Beginners_Guide/traceiosect.html
If anybody else would like to find out (and share) what their stats look like that would be helpful.
[details=Try this]install
apt-get update && apt-get install sysstat iotop -y
Uptime, total write to partition, writes by dnsmasq, followed by pihole-ftl
uptime && iostat -m && cat /proc/$(pidof dnsmasq)/io && echo && cat /proc/$(pidof pihole-FTL)/io
ls -lh /etc/pihole/pihole-FTL.db /var/log/pihole-FTL.log /var/log/pihole.log
Keep running to see top writes
iotop -a[/details]
