Filter the Tail PiHole Log

i'd love this, i use the tail log feature all the time, it might be my most-used pi-hole function. being able to filter it to one or multiple devices would be amazing.

the web interface already features this in the wuery log. if you are not able to use the query log due to the php iosseus try pihoe checkout dev and try the FTL branch

this is for the purpose of watching new requests in real time, not seeing past queries

1 Like

Actually I added the Pi-hole tail log function a really long time ago to help me debugging stuff, but at some point I thought that others might want to have this as well and here we are: I was correct :slight_smile:

As you may or not have already seen in the code, this is merely a pure tail -f /var/log/pihole.log (a little more fancy than that) implementation, so I'm not sure where to apply the filtering.

If you insist on regular expressions the best way might be to implement it in the displaying browser using some JavaScript. What do you think?

That makes it easier (KISS = Keep It Simple Stupid!)
So we just need two dialog boxes, one for the search parameter and the other a tick box for if we want to use regex.
And just do a:

tail -f /var/log/pihole.log | grep -e REGEX

This is itching so am going to look into this if can do myself :wink:

1 Like

Unfortunately it is not as easy as using tail -f in the command line, since I had to ensure that is also works over network connections that are not reliable (hence it was likely that the connection gets interrupted).

Have a look here for my actual implementation:

Initially, I ask for the position of the end of the file and then I only ask for updates and update the front end accordingly.

The regex could be applied on the PHP level (after calling fgets()), but then it would have to be validates/sanitized. A solution inside the clients browser still appears to be the simplest way.

I second this.

If you can't filter the pihole.log, then would it be possible to add a live view feature to specific IP addresses? Instead of having to click on a specific IP to see it's connections(successful and blocked) and then having to refresh the page to see new connections since, just make that page do a live view of the connections to that specific IP.

Hi, greetings from 2019.

Wanted to request something like this but seems like -t was designed for debugging by the developer.

I'm hoping to be able to Filter by Host (Who) and See the response as well (Blocked or Replied).

tail | grep doesnt seem to be able to do that... Any help?

1 Like

You will need to use the live tail command. The command below will show in real time all log lines containing that IP.

tail -f /var/log/pihole.log | grep 192.168.0.135

Apologies for my laziness.. my problem is not the tail command...

What I like to "grep" is the response (line 2 below) in addition to the client. In essence, "tail log filter by client".

Nov 25 10:48:52 dnsmasq[550]: query[A] www.googleadservices.com from 192.168.1.1
Nov 25 10:48:52 dnsmasq[550]: /etc/pihole/gravity.list www.googleadservices.com is 0.0.0.0

This information is presented in the Pi-Hole query log on the web admin GUI. On each line, you see the time, type of query, requested domain, requesing client, status and reply. If you click on the client, the output filters on that client. Is this not the information and format you are looking for?

Matching lines in /var/log/pihole.log for this query

Nov 25 04:12:17 dnsmasq[723]: query[A] www.googleadservices.com from 127.0.0.1

Nov 25 04:12:17 dnsmasq[723]: /etc/pihole/gravity.list www.googleadservices.com is 0.0.0.0

Yes I know that's there. Appreciate the help.

It would be nice to have it "live" using tail to look at a specific client in a noisy (many clients) network. Especially the colour coding for blocked/allowed queries.

My use case would be.. Eg: monitor my phone for apps/services that "calls home"... (non-rooted)

but hey, i know this is an extremely low priority thing. it's just a nice to have in a noisy network.

If "live" is not as important for your task (and I don't think it is!) and you can live with up to one minute of delay - just query the database. FTL does a lot of work to store exactly the data you are looking for in a ready-to-use format for you!

Try

sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM queries WHERE client = '127.0.0.1' ORDER BY id DESC LIMIT 10;"

This will give you the most recent 10 queries for the client with the IP address 127.0.0.1. The returned fields are described here: Redirecting...

You should be able to easily process this data with any suitable tool.

Of course, you can also narrow down the response further when specifying more WHERE restrictions, for instance

sqlite3 /etc/pihole/pihole-FTL.db "SELECT * FROM queries WHERE client = '127.0.0.1' AND (status = 2 OR status = 3) ORDER BY id DESC LIMIT 10;"

to get only permitted (as in: not blocked) queries from the database.

We're here to assist you with any questions you may have.

1 Like

Hi,

Thought I drop this here for people who want this.

Purpose: Highlight text provided by user for pihole -t
Example: pihole -t [user_text]

The user_text will be in colored in output text. DO NOTE that this is a simple implementation, hence there is no format checking so [user_text] would work with any given text.

Step 1: Edit /usr/local/bin/pihole

Step 2: Replace tailFunc() with below

tailFunc() {
  # Warn user if Pi-hole's logging is disabled
  local logging_enabled=$(grep -c "^log-queries" /etc/dnsmasq.d/01-pihole.conf)
  if [[ "${logging_enabled}" == "0" ]]; then
    # No "log-queries" lines are found.
    # Commented out lines (such as "#log-queries") are ignored
    echo "  ${CROSS} Warning: Query logging is disabled"
  fi
  echo -e "  ${INFO} Press Ctrl-C to exit"

  # Retrieve IPv4/6 addresses
  source /etc/pihole/setupVars.conf

  # Strip date from each line
  # Colour blocklist/blacklist/wildcard entries as red
  # Colour A/AAAA/DHCP strings as white
  # Colour everything else as gray
  # Colour user provided text if available
  if [[ $2 == "" ]]; then
    tail -f /var/log/pihole.log | sed -E \
      -e "s,($(date +'%b %d ')| dnsmasq\[.*[0-9]\]),,g" \
      -e "s,(.*(blacklisted |gravity blocked ).* is (0.0.0.0|::|NXDOMAIN|${IPV4_ADDRESS%/*}|${IPV6_ADDRESS:-NULL}).*),${COL_RED}&${COL_NC}," \
      -e "s,.*(query\\[A|DHCP).*,${COL_NC}&${COL_NC}," \
      -e "s,.*,${COL_GRAY}&${COL_NC},"
  else
    tail -f /var/log/pihole.log | sed -E \
      -e "s,$2,${COL_GREEN}&${COL_NC}," \
      -e "s,($(date +'%b %d ')| dnsmasq\[.*[0-9]\]),,g" \
      -e "s,(.*(blacklisted |gravity blocked ).* is (0.0.0.0|::|NXDOMAIN|${IPV4_ADDRESS%/*}|${IPV6_ADDRESS:-NULL}).*),${COL_RED}&${COL_NC}," \
      -e "s,.*(query\\[A|DHCP).*,${COL_NC}&${COL_NC}," \
      -e "s,.*,${COL_GRAY}&${COL_NC},"
  fi
  exit 0
}

Step 3: Add $@ to function handler

  "-t" | "tail"                 ) tailFunc "$@";;

Sample: pihole -t 10.8.0.3

Annotation 2020-02-12 164800

2 Likes

Don't do this unless you know that you will not be able to update Pi-hole anymore.

Thanks and noted. So a checkout to revert to orig should fix any broken updates, yes?

put the script in your .bash_aliases and you can update as normal
also worked on this Not sophisticated but it works

#!/usr/bin/env python3

from colorama import Fore

log = open('/var/log/pihole.log', 'r')
search = input('Enter ip: ')

while True:
    for line in log:
        if 'query' in line and search in line:
            print(Fore.RESET + '-------------------------------------')
            print(Fore.GREEN + line)
            for line in log:
                if ('query' in line and search in line):
                    print(Fore.RESET + '-------------------------------------')
                    print(line)
                    break
                elif 'query' in line and search not in line:
                    break
                elif 'forwarded' in line or 'reply' in line or 'cached' in line or 'NXDOMAIN' in line:
                    print(Fore.YELLOW + line)
                    break
                elif 'blocked' in line:
                    print(Fore.RED + line)
                    break
                else:
                    print(line)
                    break

This PR will add the possibility to use pihole -t [arg] to grep for arg at the command line and preserve the colored output

This has been implemented in the latest version of Pi-hole