Lists can't add identical url twice (blocklist / allowlist)

jpgpi250 · February 11, 2024, 2:03pm

Assume the following scenario:
Administrator wants to block twitter, using this list for all users (blocklist), but wants to ensure twitter is fully reachable for users in a specific group, by assigning the same list (allowlist) to a specific group (allow_twitter).
To ensure this works, the lists needs to be entered as blocklist ( -> gravity) and as allowlist (-> antigravity). Currently, you can't add both entries, see error.

See also the identical problem and solution in this FTL change.

DL6ER · February 11, 2024, 3:59pm

I will be honest: I had to read your proposal twice but now I do agree. I also agree that the solution would be the same as in the PR you linked where it was allowed for domains on the domainlist table.

Please check if

pihole checkout core tweak/allow_adlist_dups

does what you expect it to do.

github.com/pi-hole/pi-hole

Allow adlist duplicates

pi-hole:development-v6 ← pi-hole:tweak/allow_adlist_dups

opened 03:59PM - 11 Feb 24 UTC

DL6ER

+3 -2

# What does this implement/fix? Change UNIQUEness constraint from (`address`)… to (`address, type`) for the `adlist` table. This will allow certain adlists to be associated to different groups. A possible scenario is an adlist meant to block a specific service (e.g. Twitter, Youtube, etc.). It can then either be used to ensure these services are really blocked on the devices of group A but will never be blocked on devices of group B. This is a feature request submitted on Discourse (see link below). --- **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

jpgpi250 · February 11, 2024, 4:50pm

web interface problem: deleting one entry results in ~~all~~ both entries being removed

pihole -q when only allowlist exists:

pi@raspberrypi:~ $ pihole -q twitter.meezaan.org
Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 1 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org

pihole -q when both allowlist and blocklist exist:

Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 1 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org
    - twitter.meezaan.org

the 'pihole -q' result doesn't indicate if the entry is on an allowlist or a blocklist:

DL6ER · February 13, 2024, 7:49am

This also extended to not being able to edit them correctly. Many more changes were necessary to implement this, please also check out the tweak/allow_adlist_dups branch on both FTL and web as well:

pihole checkout ftl tweak/allow_adlist_dups
pihole checkout web tweak/allow_adlist_dups

(FTL binaries still building...)

The API returns the correct result

but the frontend currently fails to interpret it correctly:

~~This is the next thing on my todo list.~~
edit Fixed and pushed:

jpgpi250 · February 13, 2024, 8:21am

sudo pihole -v
Core
    Version is v5.17.3-285-g75fadb9b (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 75fadb9b (Latest: 75fadb9b)
Web
    Version is v5.19-675-g2098a11a (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 2098a11a (Latest: 2098a11a)
FTL
    Version is vDev-7a919cb (Latest: null)
    Branch is tweak/allow_adlist_dups
    Hash is 7a919cbf (Latest: 7a919cbf)

deleting, using web interface -> SOLVED
web interface / tools / search lists -> SOLVED, output is very informative
pihole -q (command line) -> both lists are identified, but the type info (block list // allow list is missing).

pihole -q twitter.meezaan.org
Found 0 domains exactly matching 'twitter.meezaan.org'.

Found 2 adlists exactly matching 'twitter.meezaan.org'.

  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org


  - https://raw.githubusercontent.com/blocklistproject/Lists/master/twitter.txt

    - twitter.meezaan.org

given the type (allow // block) can be retrieved, it should be possible to display this info on the command line (pihole -q)

thanks for your time and effort.

DL6ER · February 13, 2024, 8:58am

DL6ER · February 13, 2024, 9:07am

github.com/pi-hole/FTL

Allow adlist duplicates

pi-hole:development-v6 ← pi-hole:tweak/allow_adlist_dups

opened 09:07AM - 13 Feb 24 UTC

DL6ER

+164 -36

# What does this implement/fix? Allow narrowing down the (ad)list type using …the (optional) query parameter `?type={allow,block}`. Accompanying change related to https://github.com/pi-hole/pi-hole/pull/5572. See reasoning therein. This is a feature request submitted on Discourse (see link below). > [!NOTE] > This pull requests requires all three Pi-hole components to be checked out > ``` > pihole checkout core tweak/allow_adlist_dups > pihole checkout web tweak/allow_adlist_dups > pihole checkout ftl tweak/allow_adlist_dups > ``` > for testing --- **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

github.com/pi-hole/web

Allow adlist duplicates

pi-hole:development-v6 ← pi-hole:tweak/allow_adlist_dups

opened 09:06AM - 13 Feb 24 UTC

DL6ER

+36 -29

# What does this implement/fix? Accompanying change related to https://github….com/pi-hole/pi-hole/pull/5572. See reasoning therein. This PR also adds disentanglement of adlist types in the context of the Search tool. This is a feature request submitted on Discourse (see link below). > [!NOTE] > This pull requests requires all three Pi-hole components to be checked out > ``` > pihole checkout core tweak/allow_adlist_dups > pihole checkout web tweak/allow_adlist_dups > pihole checkout ftl tweak/allow_adlist_dups > ``` > for testing --- **Related issue or feature (if applicable):** N/A **Pull request in [docs](https://github.com/pi-hole/docs) with documentation (if applicable):** N/A --- **By submitting this pull request, I confirm the following:** 1. I have read and understood the [contributors guide](https://docs.pi-hole.net/guides/github/contributing/), as well as this entire template. I understand which branch to base my commits and Pull Requests against. 2. I have commented my proposed changes within the code. 3. I am willing to help maintain this change if there are issues with it later. 4. It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) 5. I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) ## Checklist: - [x] The code change is tested and works locally. - [x] I based my code and PRs against the repositories `developmental` branch. - [x] I [signed off](https://docs.pi-hole.net/guides/github/how-to-signoff/) all commits. Pi-hole enforces the [DCO](https://docs.pi-hole.net/guides/github/dco/) for all contributions - [x] I [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) all my commits. Pi-hole requires signatures to verify authorship - [x] I have read the above and my PR is ready for review.

jpgpi250 · February 13, 2024, 9:54am

DL6ER · March 2, 2024, 5:29pm

The feature has been merged into development-v6

jpgpi250 · March 3, 2024, 8:35am

During pihole -up, there is a message:

  [i] Upgrading gravity database from version 18 to 19

In /etc/.pihole/advanced/Scripts/database_migration/gravity, there is a file 18_to_19.sql

however

pihole-FTL sqlite3 "/etc/pihole/gravity.db" ".timeout = 2000"  "SELECT value FROM 'info' WHERE property = 'version';"

returns version 18

something wrong here?

DL6ER · March 3, 2024, 9:40am

I cannot reproduce this myself. Are there any other errors printed around

?

Looking at the code, I do not see an error, if above's message is printed, the database should have been upgraded
I do get 19 myself when running the exact sqlite3 command you posted on my system - are all your components back on development-v6 ?
Is the old version still shown after you run pihole -g manually?

jpgpi250 · March 3, 2024, 10:12am

  [i] Upgrading gravity database from version 18 to 19
  [i] Restarting services...
  [i] Enabling pihole-FTL service to start on reboot...
  [i] Restarting pihole-FTL service...
  [i] Deleting existing list cache...
  [i] Neutrino emissions detected...
  [i] Storing gravity database in /etc/pihole/gravity.db

  [] Pulling blocklist source list into range

  [i] Preparing new gravity database...
  [i] Creating new gravity databases...

Core
    Version is v5.17.3-295-g0635ea74 (Latest: null)
    Branch is development-v6
    Hash is 0635ea74 (Latest: 0635ea74)
Web
    Version is v5.19-678-g64e865e7 (Latest: null)
    Branch is development-v6
    Hash is 64e865e7 (Latest: 64e865e7)
FTL
    Version is vDev-7ef8e6c (Latest: null)
    Branch is development-v6
    Hash is 7ef8e6c9 (Latest: 7ef8e6c9)

pihole -g
  [i] Upgrading gravity database from version 18 to 19

pihole-FTL sqlite3 "/etc/pihole/gravity.db" ".timeout = 2000" "SELECT value FROM 'info' WHERE property = 'version';"
18

it appears to be possible to add an allowlist and blocklist (same url), so the database appears to be updated.