Lists can't add identical url twice (blocklist / allowlist)

Assume the following scenario:
Administrator wants to block twitter, using this list for all users (blocklist), but wants to ensure twitter is fully reachable for users in a specific group, by assigning the same list (allowlist) to a specific group (allow_twitter).
To ensure this works, the lists needs to be entered as blocklist ( -> gravity) and as allowlist (-> antigravity). Currently, you can't add both entries, see error.

See also the identical problem and solution in this FTL change.

I will be honest: I had to read your proposal twice but now I do agree. I also agree that the solution would be the same as in the PR you linked where it was allowed for domains on the domainlist table.

Please check if

pihole checkout core tweak/allow_adlist_dups

does what you expect it to do.

web interface problem: deleting one entry results in all both entries being removed

pihole -q when only allowlist exists:

pi@raspberrypi:~ $ pihole -q twitter.meezaan.org
Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 1 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org

pihole -q when both allowlist and blocklist exist:

Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 1 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org
    - twitter.meezaan.org

the 'pihole -q' result doesn't indicate if the entry is on an allowlist or a blocklist:

This also extended to not being able to edit them correctly. Many more changes were necessary to implement this, please also check out the tweak/allow_adlist_dups branch on both FTL and web as well:

pihole checkout ftl tweak/allow_adlist_dups
pihole checkout web tweak/allow_adlist_dups

(FTL binaries still building...)

The API returns the correct result


but the frontend currently fails to interpret it correctly:
Screenshot from 2024-02-13 08-46-33

This is the next thing on my todo list.
edit Fixed and pushed:

sudo pihole -v
Core
    Version is v5.17.3-285-g75fadb9b (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 75fadb9b (Latest: 75fadb9b)
Web
    Version is v5.19-675-g2098a11a (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 2098a11a (Latest: 2098a11a)
FTL
    Version is vDev-7a919cb (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 7a919cbf (Latest: 7a919cbf)
  • deleting, using web interface -> SOLVED
  • web interface / tools / search lists -> SOLVED, output is very informative
  • pihole -q (command line) -> both lists are identified, but the type info (block list // allow list is missing).
pihole -q twitter.meezaan.org
Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 2 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org


  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org

given the type (allow // block) can be retrieved, it should be possible to display this info on the command line (pihole -q)

thanks for your time and effort.

:+1:

The feature has been merged into development-v6

During pihole -up, there is a message:

  [i] Upgrading gravity database from version 18 to 19

In /etc/.pihole/advanced/Scripts/database_migration/gravity, there is a file 18_to_19.sql

however

pihole-FTL sqlite3 "/etc/pihole/gravity.db" ".timeout = 2000"  "SELECT value FROM 'info' WHERE property = 'version';"

returns version 18

something wrong here?

I cannot reproduce this myself. Are there any other errors printed around

?

  1. Looking at the code, I do not see an error, if above's message is printed, the database should have been upgraded
  2. I do get 19 myself when running the exact sqlite3 command you posted on my system - are all your components back on development-v6 ?
  3. Is the old version still shown after you run pihole -g manually?
  [i] Upgrading gravity database from version 18 to 19
  [i] Restarting services...
  [i] Enabling pihole-FTL service to start on reboot...
  [i] Restarting pihole-FTL service...
  [i] Deleting existing list cache...
  [i] Neutrino emissions detected...
  [i] Storing gravity database in /etc/pihole/gravity.db

  [] Pulling blocklist source list into range

  [i] Preparing new gravity database...
  [i] Creating new gravity databases...
Core
    Version is v5.17.3-295-g0635ea74 (Latest: null)
    Branch is development-v6
    Hash is 0635ea74 (Latest: 0635ea74)
Web
    Version is v5.19-678-g64e865e7 (Latest: null)
    Branch is development-v6
    Hash is 64e865e7 (Latest: 64e865e7)
FTL
    Version is vDev-7ef8e6c (Latest: null)
    Branch is development-v6
    Hash is 7ef8e6c9 (Latest: 7ef8e6c9)
pihole -g
  [i] Upgrading gravity database from version 18 to 19
pihole-FTL sqlite3 "/etc/pihole/gravity.db" ".timeout = 2000" "SELECT value FROM 'info' WHERE property = 'version';"
18

it appears to be possible to add an allowlist and blocklist (same url), so the database appears to be updated.