Encountered non-critical SQL warnings

General Beta 5.0
27

How about just not load the list and show in the gravity output that the list was skipped? Then the user knows which list is a problem and can go fix or delete the problem. Similar to what is done now with adBlock formatted lists - they are identified and skipped.

1 Like
28

That's something to look at as well. I think users are just going to ignore the gravity output and complain that the blocked domain numbers are wrong. But we may have that no matter what.

29

Sorry, I'm NOT a programmer, but isn't the code already there? pihole -g now checks the result of ALL lists. Is it impossible to use the same routine to check a single list, don't write to the database (command line option checklist), and output the result on screen?

30

The main issue is that linting lists in bash is insanely stupid and problematic. Things sqlite3 can catch, we cant. I don't see any real way to check other than to try and push it in, and report if it fails. I don't think we can report why it fails with any kind of reliability.

31

command line option checklist, using the code from pihole -g, processing a single list, write to checklist.db, catch the errors.

32

This is mostly it, and it's a nightmare to maintain. Expanding on it is so rife with the possibility of massive screwups.

    < ${source} awk -F '#' '{print $1}' | \
    awk -F '/' '{print $1}' | \
    awk '($1 !~ /^#/) { if (NF>1) {print $2} else {print $1}}' | \
    sed -nr -e 's/\.{2,}/./g' -e '/\./p' >  ${destination}
    return 0
  fi

    awk '
      # Remove URL scheme, optional "username:password@", and ":?/;"
      # The scheme must be matched carefully to avoid blocking the wrong URL
      # in cases like:
      #   http://www.evil.com?http://www.good.com
      # See RFC 3986 section 3.1 for details.
      /[:?\/;]/ { gsub(/(^[a-zA-Z][a-zA-Z0-9+.-]*:\/\/(.*:.*@)?|[:?\/;].*)/, "", $0) }
      # Skip lines which are only IPv4 addresses
      /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/ { next }
      # Print if nonempty
      length { print }
    ' "${source}" 2> /dev/null > "${destination}"
33

Sounds easy, it's not.

1 Like
34

Pi-Hole should not become a tool to fix, clean up, unscrew, etc. dodgy lists. HOSTS format is a very simple concept, and it's not hard to find or make lists that meet this format standard. If they don't meet that standard (and you can quickly look at them to check), don't use them.

35

OK, I rest my case, out of arguments for this.

please consider (OR, not AND):

  • keep the tmp file if possible.
  • output the listname with the error message, or at least the original string, that caused the error.
1 Like
37

I'm willing to implement something when someone can tell me the awk command that separates

/tmp/tmp.yrjMjpYGzI.gravity:1866856: expected 2 columns but found 3 - extras ignored
/tmp/tmp.yrjMjpYGzI.gravity:1866857: expected 2 columns but found 3 - extras ignored
/tmp/tmp.yrjMjpYGzI.gravity:1866858: expected 2 columns but found 3 - extras ignored

automatically into something over which I can run

cat ${file} | awk 'NR==${line}'

Obviously, this awk (can also be something else) needs to be able to skip a line which does not start in

file:line: message

edit Not very elegant but working way:

echo "/tmp/tmp.yrjMjpYGzI.gravity:1866856: expected 2 columns but found 3 - extras ignored" | grep -oh "^/[^:]*"
/tmp/tmp.yrjMjpYGzI.gravity

echo "/tmp/tmp.yrjMjpYGzI.gravity:1866856: expected 2 columns but found 3 - extras ignored" | grep -oh ":[^:]*:" | sed "s/://g;"
1866856

edit 2 Somewhat better

var="$(echo "/tmp/tmp.yrjMjpYGzI.gravity:1866856: expected 2 columns but found 3 - extras ignored" | grep -oh "^[^:]*:[0-9]*")"

echo "${var}"
/tmp/tmp.yrjMjpYGzI.gravity:1866856

echo "${var#*:}"
1866856

echo "${var%:*}"
/tmp/tmp.yrjMjpYGzI.gravity
1 Like
39

Does sqlite3 give a non-zero exit code when this happens? If so we can use that and report that things failed.

I don't think we can say anything other than the list failed to import.

40

No. It is a non-critical warning resulting in return code 0.

43

Was out of arguments, but now you're giving me on. You both say you can show the list name in the error message, why don't you?

44

head -n "${var#*:}" "${var%:*}" | tail -n 1 ?

45

He was proposing a hypothetical. Not stating that it could be done.

46

Final question, if I get a single error message, does this mean a single line of a list isn't used OR is the entire list not used?

47

I don't know offhand but my view is that a single error taints the entire list and the whole thing needs to be tossed.

2 Likes
48

Entirely untested (because I'm too tired by now) but it may work. At least, it cannot cause any harm, it would just not work if it doesn't work. If someone wants to try it:

pihole checkout core tweak/database_warnings_inspection

This is a once for each domain error. It typically mean that one domain is not imported. Maybe two if the format is really screwed up. It would be helpful if you could run this as a test as you're currently seeing a warning whereas I'm not seeing one with the stock lists @jpgpi250.

2 Likes
49 
  : Could not update local repository. Contact support.
   Error: Unable to complete update, please contact support
50

Yeah I think you need to do something. I'm fairly new to the pihole forums but it looks like some people are overzealous with the number and quality of the lists they choose to use, not knowing they are not actually compatible.
Might save the devs some time in the forums if a line in the sand is drawn and junk lists are simply ignored and flagged - if possible in the web interface, which caters for all users... else a pihole cli or even web gui check where users can easily check on their lists for compatibility.

2 Likes