This one posted (the problem was that the log directory was full before, and your local copy is saved first).
Let's figure out why your log is filling up so quickly. Please post the outputs of the following commands from the Pi terminal:
echo ">stats >quit" | nc localhost 4711
echo ">top-clients >quit" | nc localhost 4711
echo ">top-domains >quit" | nc localhost 4711
echo ">top-ads >quit" | nc localhost 4711
Likely unrelated to your problem, but noted in your debug log. You have a number of regex entries that won't work - https is not part of the domain name.
*** [ DIAGNOSING ]: Domainlist (0/1 = exact white-/blacklist, 2/3 = regex white-/blacklist)
id type enabled group_ids domain date_added date_modified comment
----- ---- ------- ------------ ---------------------------------------------------------------------------------------------------- ------------------- ------------------- --------------------------------------------------
...
13 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?clicks\.beap\.bc\.yahoo\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:32
14 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?secure\.footprint\.net/ 2022-05-23 23:58:22 2022-09-04 14:28:32
15 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?match\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:32
16 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?clicks\.beap\.bc\.yahoo(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
17 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?sitescout(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
18 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?appnexus(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
19 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?evidon(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
20 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?mediamath(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
21 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?scorecardresearch(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
22 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?doubleclick(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
23 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?flashtalking(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
24 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?turn(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
25 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?mathtag(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
26 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?googlesyndication(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:32
27 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?s\.yimg\.com/cv/ae/us/audience/ 2022-05-23 23:58:22 2022-09-04 14:28:33
28 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?clicks\.beap/ 2022-05-23 23:58:22 2022-09-04 14:28:33
29 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?.doubleclick(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:33
30 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?yieldmanager(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:33
31 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?w55c(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:33
32 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?adnxs(\.\w{2}\.\w{2}|\.\w{2,4})/ 2022-05-23 23:58:22 2022-09-04 14:28:33
33 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?advertising\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
34 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?evidon\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
35 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?scorecardresearch\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
36 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?flashtalking\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
37 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?turn\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
38 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?mathtag\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
39 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?surveylink/ 2022-05-23 23:58:22 2022-09-04 14:28:33
40 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?info\.yahoo\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
41 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?ads\.yahoo\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
42 3 1 0 ^https?://([A-Za-z0-9.-]*\.)?global\.ard\.yahoo\.com/ 2022-05-23 23:58:22 2022-09-04 14:28:33
And, the regex are improperly formed as well:
pihole-FTL regex-test clicks.yahoo.com ^https?://([A-Za-z0-9.-]*\.)?clicks\.beap\.bc\.yahoo\.com/
-bash: syntax error near unexpected token `('
Thanks for pointing this out. I was thinking that there is nearly no benefit of only 0-1h of logs when RAM logging is enabled (not allowing statistical analysis), but actually for debugging recent DNS queries from console it has still some use.
The tail looks like they are actually not filling fast, but /var/log was never cleared because cron.service was masked, breaking also Pi-hole's own log rotation and update checks.
Makes sense. In a normal (not ram logged and daily cron) install, you would typically see the previous several days logs:
ls -lha /var/log/pihole
total 5.0M
drwxr-xr-x 2 pihole pihole 4.0K Sep 17 00:00 .
drwxr-xr-x 9 root root 4.0K Sep 17 00:00 ..
-rw-r--r-- 1 pihole pihole 1.7K Sep 17 08:30 FTL.log
-rw-r--r-- 1 pihole pihole 4.7K Sep 17 00:00 FTL.log.1
-rw-r--r-- 1 pihole pihole 828 Sep 16 00:00 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 3.7K Sep 15 00:00 FTL.log.3.gz
-rw-r--r-- 1 root root 0 Jul 7 23:01 gravity.db
-rw-r----- 1 root pihole 40K Sep 14 16:42 pihole_debug.log
lrwxrwxrwx 1 pihole pihole 23 Jul 7 15:45 pihole-FTL.log -> /var/log/pihole/FTL.log
-rw-r----- 1 pihole pihole 1.1M Sep 17 08:36 pihole.log
-rw-r----- 1 pihole pihole 3.0M Sep 17 00:00 pihole.log.1
-rw-r----- 1 pihole pihole 292K Sep 16 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 253K Sep 15 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 155K Sep 14 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 101K Sep 13 00:00 pihole.log.5.gz
-rw-r--r-- 1 root root 1.5K Sep 11 04:39 pihole_updateGravity.log
The OP shows a dnsmasq log that is several days old:
total 49M
drwxr-xr-x 2 pihole pihole 100 Sep 15 11:53 .
drwxr-xr-x 6 root root 260 Sep 4 14:11 ..
-rw-r--r-- 1 pihole pihole 52K Sep 17 16:11 FTL.log
-rw-r----- 1 pihole pihole 49M Sep 13 19:18 pihole.log
-rw-r----- 1 root pihole 0 Sep 17 16:10 pihole_debug.log
Even at that, 50 MB in 4 days is a decent amount of log activity.
sudo systemctl unmask cron
sudo systemctl restart cron
Ran both of these.
ls -Al /boot/dietpi
total 1168
-rwxr-xr-x 1 root root 1081 Sep 4 13:55 .dietpi-backup_inc_exc
-rwxr-xr-x 1 root root 577 Sep 4 13:55 .dietpi-services_include_exclude
-rwxr-xr-x 1 root root 361 Sep 4 14:26 .hw_model
-rwxr-xr-x 1 root root 2 Sep 4 14:01 .install_stage
-rwxr-xr-x 1 root root 15 Jul 31 23:05 .prep_info
-rwxr-xr-x 1 root root 112 Sep 4 14:01 .version
-rwxr-xr-x 1 root root 11511 Aug 29 05:51 dietpi-autostart
-rwxr-xr-x 1 root root 24254 Aug 29 05:51 dietpi-backup
-rwxr-xr-x 1 root root 7840 Aug 29 05:51 dietpi-bugreport
-rwxr-xr-x 1 root root 14521 Aug 29 05:51 dietpi-cleaner
-rwxr-xr-x 1 root root 43539 Aug 29 05:51 dietpi-cloudshell
-rwxr-xr-x 1 root root 140799 Aug 29 05:51 dietpi-config
-rwxr-xr-x 1 root root 5262 Aug 29 05:51 dietpi-cpuinfo
-rwxr-xr-x 1 root root 9386 Aug 29 05:51 dietpi-cron
-rwxr-xr-x 1 root root 16699 Aug 29 05:51 dietpi-ddns
-rwxr-xr-x 1 root root 74205 Aug 29 05:51 dietpi-drive_manager
-rwxr-xr-x 1 root root 7219 Aug 29 05:51 dietpi-explorer
-rwxr-xr-x 1 root root 3960 Aug 29 05:51 dietpi-launcher
-rwxr-xr-x 1 root root 4991 Aug 29 05:51 dietpi-led_control
-rwxr-xr-x 1 root root 21334 Aug 29 05:51 dietpi-letsencrypt
-rwxr-xr-x 1 root root 9003 Aug 29 05:51 dietpi-login
-rwxr-xr-x 1 root root 6789 Aug 29 05:51 dietpi-morsecode
-rwxr-xr-x 1 root root 35339 Aug 29 05:51 dietpi-services
-rwxr-xr-x 1 root root 674027 Aug 29 05:51 dietpi-software
-rwxr-xr-x 1 root root 7064 Aug 29 05:51 dietpi-survey
-rwxr-xr-x 1 root root 17082 Aug 29 05:51 dietpi-sync
-rwxr-xr-x 1 root root 22531 Aug 29 05:51 dietpi-update
-rwxr-xr-x 1 root root 20985 Aug 29 05:51 dietpi-vpn
drwxr-xr-x 2 root root 2048 Aug 29 05:51 func
drwxr-xr-x 2 root root 512 Aug 29 05:51 misc
-rwxr-xr-x 1 root root 2426 Aug 29 05:51 postboot
-rwxr-xr-x 1 root root 1393 Aug 29 05:51 preboot
You mean you do run
pihole -ton the console or parse/var/log/pihole/pihole.logmanually by times? Then I think RAM logging isn't suitable, since you'll only have one hour of logs in this file (not affecting the web UI/dashboard), when clearing logs works as intended.
No, I use the web dashboard and use the query log to quick add new domains to the blacklist.
Let's figure out why your log is filling up so quickly. Please post the outputs of the following commands from the Pi terminal:
echo ">stats >quit" | nc localhost 4711
domains_being_blocked 884886
dns_queries_today 11903
ads_blocked_today 2258
ads_percentage_today 18.970007
unique_domains 1356
queries_forwarded 9074
queries_cached 565
clients_ever_seen 9
unique_clients 9
dns_queries_all_types 11903
reply_UNKNOWN 39
reply_NODATA 2387
reply_NXDOMAIN 2018
reply_CNAME 3219
reply_IP 4107
reply_DOMAIN 24
reply_RRNAME 4
reply_SERVFAIL 0
reply_REFUSED 0
reply_NOTIMP 0
reply_OTHER 0
reply_DNSSEC 0
reply_NONE 0
reply_BLOB 105
dns_queries_all_replies 11903
privacy_level 0
status enabled
echo ">top-clients >quit" | nc localhost 4711
0 4550 192.168.10.195
1 3547 192.168.10.52
2 2107 192.168.10.78
3 1255 192.168.10.44
4 405 192.168.10.66
5 83 127.0.0.1 localhost
6 2 192.168.10.1
7 1 192.168.10.25 pi.hole
8 1 ::1 localhost
9 1 fe80::e65f:1ff:feaf:cc29 pi.hole
echo ">top-domains >quit" | nc localhost 4711
0 1749 lb._dns-sd._udp.0.10.168.192.in-addr.arpa
1 508 gateway.fe.apple-dns.net
2 180 imap.gmail.com
3 157 chat.signal.org
4 134 guzzoni-apple-com.v.aaplimg.com
5 123 trello.com
6 123 mail.google.com
7 115 www.google.com
8 109 mmx-ds.cdn.whatsapp.net
9 104 ocsp.pki.goog
echo ">top-ads >quit" | nc localhost 4711
0 567 gs-loc.apple.com
1 166 self.events.data.microsoft.com
2 147 dit.whatsapp.net
3 129 telemetry-in.battle.net
4 97 ocsp.usertrust.com
5 90 mask.icloud.com
6 63 tsfe.trafficshaping.dsp.mp.microsoft.com
PS: I turned off logging recently and flushed the logs so its no longer full but yeah I too would like to know why its filling up so soon.
I ran ls -lha /var/log/pihole and again you can see pihole.log filling up really quickly.
total 4.0M
drwxr-xr-x 2 pihole pihole 100 Sep 15 11:53 .
drwxr-xr-x 6 root root 260 Sep 4 14:11 ..
-rw-r--r-- 1 pihole pihole 75K Sep 18 02:26 FTL.log
-rw-r----- 1 pihole pihole 3.8M Sep 18 03:24 pihole.log
-rw-r----- 1 root pihole 87K Sep 17 16:27 pihole_debug.log
Another thing, I added one more computer (a windows computer) onto the pihole network. Is there any reason why a windows gaming computer would fill up the log quickly?
Ok.. ill just remove all the ones with HTTPS for now. I copied these off a regex blacklist post on here.
Then I'd say you do not need this file query logging. However, with logs being cleared hourly it wouldn't hurt either.
Your /boot/dietpi content shows that .installed is missing, which should be present if first run setup was finished one time. Could you show:
cat /boot/dietpi/.install_stage
If it shows 0 or 1, please run dietpi-software and simply select "Install". If it shows 2, please run dietpi-software install 103, which reconfigures the RAM disk, being a noop, but also sets the logging choice variable in /boot/dietpi/.installed for the hourly cron job to clear logs. Aside of the masked cron service, this is a second reason why currently logs are not cleared.
Are you sure that this added computer is causing the logs? From the top clients stats (which are coming from the database, not from the log file), no client has an exceptionally high number of queries. But yeah, finding the culprit in /var/log/pihole/pihole.log is key.
cat /boot/dietpi/.install_stage
Yeah its 1
If it shows
0or1, please rundietpi-softwareand simply select "Install".
Is it a fresh install? Will I loose all blacklists etc?
Are you sure that this added computer is causing the logs? From the top clients stats (which are coming from the database, not from the log file), no client has an exceptionally high number of queries. But yeah, finding the culprit in
/var/log/pihole/pihole.logis key.
No I don't think that, was just sharing I had a new device connected to the pihole.
If it shows
0or1, please rundietpi-softwareand simply select "Install".
I ran it and now set log system to DietPi-RAMlog #1 which is hourly clear. Does this mean that my query logs from the web UI will flushed hour or hour?
After the setup I again ran cat /boot/dietpi/.install_stage and when I got a response 2 I ran dietpi-software install 103 which gave me the below.
[ OK ] DietPi-Software | Initialised database
[ OK ] DietPi-Software | Reading database
DietPi-Software
─────────────────────────────────────────────────────
Mode: Automated install
[ INFO ] DietPi-Software | 103: DietPi-RAMlog is already installed
[ INFO ] DietPi-Software | Use "dietpi-software reinstall 103" to force rerun of installation and configuration steps for DietPi-RAMlog.
[ OK ] DietPi-Software | No changes applied for: DietPi-RAMlog
Does it all look ok now?
Just ran sudo systemctl status cron and its definitely changed..
● cron.service - Regular background program processing daemon
Loaded: loaded (/lib/systemd/system/cron.service; disabled; vendor preset: enabled)
Active: active (running) since Mon 2022-09-19 11:50:30 +08; 7min ago
Docs: man:cron(8)
Main PID: 8222 (cron)
Tasks: 1 (limit: 4915)
CPU: 11ms
CGroup: /system.slice/cron.service
└─8222 /usr/sbin/cron -f
Sep 19 11:50:30 DietPi systemd[1]: Started Regular background program processing daemon.
Sep 19 11:50:30 DietPi cron[8222]: (CRON) INFO (pidfile fd = 3)
Sep 19 11:50:30 DietPi cron[8222]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
and ran ls -lha /var/log/pihole
total 14M
drwxr-xr-x 2 pihole pihole 120 Sep 19 00:00 .
drwxr-xr-x 6 root root 260 Sep 4 14:11 ..
-rw-r--r-- 1 pihole pihole 98K Sep 19 11:56 FTL.log
-rw-r----- 1 pihole pihole 2.9M Sep 19 11:59 pihole.log
-rw-r----- 1 pihole pihole 11M Sep 19 00:00 pihole.log.1
-rw-r----- 1 root pihole 87K Sep 17 16:27 pihole_debug.log
I'll keep a close watch on errors for the next few days.
Thank you so much for your help with this.
Thank you so much for your help and inputs on this.
No, it does not touch any installed software. It's the first run setup dialog. Actually it should show up on login as long as /boot/dietpi/.install_stage is not set to 2, so I'm not 100% sure how you could have skipped/missed it 
.
Not the ones from the web UI, only the ones from the pihole.log. By default, Pi-hole logs DNS queries redundant to two places: /var/log/pihole/pihole.log and /etc/pihole/pihole-FTL.db. Only the second is used by the web UI, the first is used when running pihole -t from console.
It does 
.
Cron looks fine now. Also Pi-hole's log rotation ran successfully already. Due to hourly cron, the log file sizes should now stay below 1 MiB. It's still large, so you may still want to check the pihole.log for something unexpected. Probably a client spams PTR (reverse DNS) requests or so.
From above it appears cron is not enabled to start at boot:
pi@ph5b:~ $ systemctl is-enabled cron.service
enabled
pi@ph5b:~ $ systemctl is-active cron.service
active
DietPi starts it at a later boot stage. An old confusing DietPi behaviour I want to get rid of for a long time 
: DietPi-Services | Do not "disable" controlled services · Issue #5470 · MichaIng/DietPi · GitHub
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.