Initial deployment docker errors

p928s1984 · June 23, 2020, 9:03pm

Hello, I am attempting my first deployment of Pi hole. looks like a great product.
I am using docker, every time I deploy, the container starts. but UI never comes up. I restart the container, UI works but I get the following error, and things are not working correctly.

Not sure what I am doing wrong, hopefully someone can help

Info:
using pihole:latest
tried running pihole -g
tried deleting the DB's and restarting.

 ::: Docker start setup complete
Error: no such table: info
  [i] Neutrino emissions detected...
  [✗] Pulling blocklist source list into range                                                                                                                                                                                                          
  [i] No source list found, or it is empty

  [i] Building tree...
  [✗] Unable to build gravity tree in /etc/pihole/gravity_temp.db
  Error: no such table: main.gravity

  [✗] Unable to update gravity timestamp in database /etc/pihole/gravity.db
  Error: near line 2: no such table: info
Error: no such table: vw_gravity
Error: no such table: vw_gravity
  [i] Number of gravity domains:  ( unique domains)
Error: near ")": syntax error
Error: no such table: vw_blacklist
  [i] Number of exact blacklisted domains: 
Error: no such table: vw_regex_blacklist
  [i] Number of regex blacklist filters: 
Error: no such table: vw_whitelist
  [i] Number of exact whitelisted domains: 
Error: no such table: vw_regex_whitelist
  [i] Number of regex whitelist filters: 
  [✓] Cleaning up stray matter

yubiuser · June 24, 2020, 7:33am

Your likely facing this issue because ot

No source list found, or it is empty

You can try to fix this by creating the file

/etc/pihole/adlists.list

and add at least one adlist like https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

then run
pihole -g -r

p928s1984 · June 24, 2020, 2:21pm

No go.

root@pi:/# cat /etc/pihole/adlist.list 
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
root@pi:/# pihole -g -r
  [✓] Restoring from migration backup                                                                          
  [i] Creating new gravity database
  [i] Migrating content of /etc/pihole/adlists.list into new database
  [i] Neutrino emissions detected...
  [✗] Pulling blocklist source list into range                                                                 
  [i] No source list found, or it is empty

  [i] Building tree...
  [✗] Unable to build gravity tree in /etc/pihole/gravity_temp.db
  Error: no such table: main.gravity
  [i] Number of gravity domains: 0 (0 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache                                                                                       
  [✓] Cleaning up stray matter                                                                                 

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

yubiuser · June 24, 2020, 5:30pm

p928s1984:

root@pi:/# cat /etc/pihole/adlist.list 
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
root@pi:/# pihole -g -r
  [✓] Restoring from migration backup                                                                          
  [i] Creating new gravity database
  [i] Migrating content of /etc/pihole/adlists.list into new database

You missed a s, it's adlists.list

jfb · June 24, 2020, 5:49pm

I think the appropriate location for the adlists.list is in the migration_backup folder. That is where the gravity recreate code obtains the lists.

github.com

pi-hole/pi-hole/blob/4d25f695267590b61a4061f9bb43448005d99b85/gravity.sh#L783


    "-f" | "--force" ) forceDelete=true;;
    "-o" | "--optimize" ) optimize_database=true;;
    "-r" | "--recreate" ) recreate_database=true;;
    "-h" | "--help" ) helpFunc;;
  esac
done
# Trap Ctrl-C
gravity_Trap
if [[ "${recreate_database:-}" == true ]]; then
  str="Restoring from migration backup"
  echo -ne "${INFO} ${str}..."
  rm "${gravityDBfile}"
  pushd "${piholeDir}" > /dev/null || exit
  cp migration_backup/* .
  popd > /dev/null || exit
  echo -e "${OVER}  ${TICK} ${str}"
fi
# Move possibly existing legacy files to the gravity database

yubiuser · June 24, 2020, 6:24pm

Good thought, but I think it doesn't matter as the content of migration_backup is just copied to ${piholeDir} (/etc/pihole in most cases)

github.com

pi-hole/pi-hole/blob/4d25f695267590b61a4061f9bb43448005d99b85/gravity.sh#L788


done
# Trap Ctrl-C
gravity_Trap
if [[ "${recreate_database:-}" == true ]]; then
  str="Restoring from migration backup"
  echo -ne "${INFO} ${str}..."
  rm "${gravityDBfile}"
  pushd "${piholeDir}" > /dev/null || exit
  cp migration_backup/* .
  popd > /dev/null || exit
  echo -e "${OVER}  ${TICK} ${str}"
fi
# Move possibly existing legacy files to the gravity database
migrate_to_database
if [[ "${forceDelete:-}" == true ]]; then
  str="Deleting existing list cache"
  echo -ne "${INFO} ${str}..."

jfb · June 24, 2020, 6:32pm

That's the point. If changes are made to a file in /etc/pihole, they are overwritten by the information from the migration_backup, which includes all the information at the time of conversion . So, make the changes in the migration_backup folder.

yubiuser · June 24, 2020, 6:56pm

As this is a docker related issue and pihole's container is not supposed to be updated (and the OP didn't mention v4 - > v5 update) the migration backup folder should be empty. Therfore is should not matter where the file is changed (or in this case created at all).

p928s1984 · June 24, 2020, 7:29pm

Indeed a typo... OOPS.
indeed also not a migration. new setup on docker.

root@pi:/# cat /etc/pihole/adlists.list 
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
root@pi:/# pihole -g -r
[i] Restoring from migration backup...rm: cannot remove '/etc/pihole/gravity.db': No such file or directory
  [✓] Restoring from migration backup                                                                          
  [i] Creating new gravity database
  [i] Migrating content of /etc/pihole/adlists.list into new database
  [i] Neutrino emissions detected...
  [✗] Pulling blocklist source list into range                                                                 
  [i] No source list found, or it is empty

  [i] Building tree...
  [✗] Unable to build gravity tree in /etc/pihole/gravity_temp.db
  Error: no such table: main.gravity
  [i] Number of gravity domains: 0 (0 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache                                                                                       
  [✓] Cleaning up stray matter                                                                                 

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

yubiuser · June 25, 2020, 7:20am

Ok, try to copy adlists.list to /etc/pihole/migration_backup and see if this helps (I personally doubt, but would be happy if convinced otherwise).

Also check file system permissions of /etc/pihole/

nanopi@nanopi:~$ ls -lha /etc/pihole/
total 346M
drwxrwxr-x  2 pihole pihole 4,0K Jun 25 09:18 .
drwxr-xr-x 94 root   root   4,0K Jun 25 08:08 ..
-rw-r--r--  1 root   root    276 Mai 12 20:02 adlists.list
-rw-r--r--  1 root   root    490 Jun 10 20:26 custom.list
-rw-r--r--  1 pihole pihole    0 Jun 25 09:12 dhcp.leases
-rw-r--r--  1 root   root    596 Jun 25 08:08 dns-servers.conf
-rw-r--r--  1 root   root     16 Jun 24 09:46 ftlbranch
-rw-r--r--  1 root   root     14 Jun 25 08:09 GitHubVersions
-rw-rw-r--  1 pihole pihole  96M Jun 25 08:09 gravity.db
-rw-r--r--  1 root   root   4,3M Jan 19 04:30 gravity.list
-rw-r--r--  1 root   root   1,1K Jun 25 08:08 install.log
-rw-r--r--  1 root   root   537K Jun 25 08:08 list.0.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   413K Jun 25 08:08 list.10.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   4,2M Jun 25 08:08 list.11.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   1,2M Jun 25 08:08 list.12.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root    15M Jun 25 08:08 list.13.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   4,0M Jun 25 08:08 list.14.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root    50K Jun 25 08:08 list.15.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   3,9K Jun 25 08:08 list.16.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   2,3K Jun 25 08:08 list.1.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   283K Jun 25 08:08 list.2.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root    22K Jun 25 08:08 list.3.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root    21K Jun 25 08:08 list.4.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root   3,8M Jun 25 08:08 list.5.www.github.developerdan.com.domains
-rw-r--r--  1 root   root   1,1M Jun 25 08:08 list.6.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root      0 Jun 25 08:08 list.7.dehakkelaar.nl.domains
-rw-r--r--  1 root   root   3,3M Jun 25 08:08 list.8.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root    21K Jun 25 08:08 list.9.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root     43 Jun 25 09:10 localbranches
-rw-r--r--  1 root   root     33 Jun 25 08:09 local.list
-rw-r--r--  1 root   root     49 Jun 25 09:10 localversions
-rw-r--r--  1 root   root    234 Jun 25 08:08 logrotate
-rw-r--r--  1 pihole pihole 2,4M Jun 25 08:08 macvendor.db
-rw-rw-r--  1 pihole root    188 Jun 25 09:12 pihole-FTL.conf
-rw-r--r--  1 pihole pihole  17M Jun 25 09:18 pihole-FTL.db
-rw-r--r--  1 pihole pihole 194M Jun  3 22:01 pihole-FTL-new_mac.db
-rw-r--r--  1 root   root    666 Jun 25 08:08 setupVars.conf
-rw-r--r--  1 root   root    666 Jun 25 08:08 setupVars.conf.update.bak

p928s1984 · June 25, 2020, 12:57pm

I decided to rebuild volumes and container. First boot, couldnt load the UI. Then i tried the above

First Boot:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] 01-resolver-resolv: applying... 
[fix-attrs.d] 01-resolver-resolv: exited 0.
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 20-start.sh: executing... 
 ::: Starting docker specific checks & setup for docker pihole/pihole
OK: Checks passed for /etc/resolv.conf DNS servers

nameserver 127.0.0.1
nameserver 1.1.1.1
options timeout:15
options attempts:5
Assigning random password: HAR8cLA6
  [i] Existing PHP installation detected : PHP version 7.0.33-0+deb9u7

  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [✓] Copying 01-pihole.conf to /etc/dnsmasq.d/01-pihole.conf                                                  
chown: cannot access '': No such file or directory
chmod: cannot access '': No such file or directory
chown: cannot access '/etc/pihole/dhcp.leases': No such file or directory
Setting password: HAR8cLA6
+ pihole -a -p HAR8cLA6 HAR8cLA6
  [✓] New password set
Using default DNS servers: 8.8.8.8 & 8.8.4.4
DNSMasq binding to default interface: eth0
Added ENV to php:
                        "PHP_ERROR_LOG" => "/var/log/lighttpd/error.log",
                        "ServerIP" => "127.0.0.1",
                        "VIRTUAL_HOST" => "pi.hole",
Using IPv4 and IPv6
::: setup_blocklists now setting default blocklists up: 
::: TIP: Use a docker volume for /etc/pihole/adlists.list if you want to customize for first boot
::: Blocklists (/etc/pihole/adlists.list) now set to:
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://mirror1.malwaredomains.com/files/justdomains
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
::: Testing pihole-FTL DNS: FTL started!
::: Testing lighttpd config: Syntax OK
::: All config checks passed, cleared for startup ...
 ::: Docker start setup complete
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range                                                                 

  [✓] Preparing new gravity database                                                                           
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful                                                                             
  [i] Received 57322 domains

  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: Retrieval successful                                                                             
  [i] Received 26853 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt

try with suggested:

root@pi:/# cat /etc/pihole/adlists.list
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
root@pi:/# ls -lha /etc/pihole/
total 92K
drwxrwxr-x  3 pihole pihole 4.0K Jun 25 07:55 .
drwxrwxr-x 53 root   root   4.0K Jun 25 07:53 ..
-rw-r--r--  1 root   root     14 Jun 25 07:54 GitHubVersions
-rw-r--r--  1 root   root     65 Jun 25 07:54 adlists.list
-rw-r--r--  1 root   root    596 Jun 25 07:53 dns-servers.conf
-rw-rw-r--  1 pihole pihole    0 May 10 13:50 gravity.db
-rw-r--r--  1 root   root      0 Jun 25 07:53 gravity_temp.db
-rw-r--r--  1 root   root      0 May 10 13:50 install.log
-rw-r--r--  1 root   root      0 May 10 13:50 list.0.raw.githubusercontent.com.domains
-rw-r--r--  1 root   root      0 May 10 13:50 list.1.mirror1.malwaredomains.com.domains
-rw-r--r--  1 root   root      0 May 10 13:50 list.2.s3.amazonaws.com.domains
-rw-r--r--  1 root   root      0 May 10 13:50 list.3.s3.amazonaws.com.domains
-rw-r--r--  1 root   root     36 Jun 25 07:53 local.list
-rw-r--r--  1 root   root     20 Jun 25 07:53 localbranches
-rw-r--r--  1 root   root     37 Jun 25 07:53 localversions
-rw-r--r--  1 root   root      0 May 10 13:50 logrotate
-rw-r--r--  1 pihole pihole    0 May 10 13:50 macvendor.db
drwxr-xr-x  2 root   root   4.0K May 10 13:50 migration_backup
-rw-r--r--  1 pihole pihole    0 May 10 13:50 pihole-FTL.conf
-rw-r--r--  1 root   root    44K Jun 25 07:55 pihole-FTL.db
-rw-r--r--  1 root   root    449 Jun 25 07:53 setupVars.conf
-rw-r--r--  1 root   root    427 Jun 25 07:53 setupVars.conf.update.bak
root@pi:/# pihole -g -r
  [✓] Restoring from migration backup                                                                          
  [i] Creating new gravity database
  [i] Migrating content of /etc/pihole/adlists.list into new database
  [i] Neutrino emissions detected...                                                                             [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database                                                                           
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful                                                                             
  [i] Received 57322 domains

  [✓] Storing downloaded domains in new gravity database                                                       
  [✓] Building tree                                                                                            
  [✓] Swapping databases                                                                                       
  [i] Number of gravity domains: 57322 (57322 unique domains)
  [i] Number of exact blacklisted domains: 0
  [i] Number of regex blacklist filters: 0
  [i] Number of exact whitelisted domains: 0
  [i] Number of regex whitelist filters: 0
  [✓] Flushing DNS cache                                                                                       
  [✓] Cleaning up stray matter                                                                                 

  [✓] DNS service is running
  [✓] Pi-hole blocking is Enabled

So it worked. why does the docker build not work out of the box? do i have a miss config? is this a bug?

yubiuser · June 25, 2020, 1:03pm

It looks like the output is truncated?

p928s1984 · June 25, 2020, 1:06pm

Negative. thats where it just stops and sits for hours. until i kill it and restart.

it appears that the below no longer exists? possibly no timeout for the default target lists? so the install never finishes?
Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt

i added the 4 default to my instance again, this is where it is stuck.

root@pi:/# pihole -g   
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range                                                                 

  [✓] Preparing new gravity database                                                                           
  [i] Target: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
  [✓] Status: Retrieval successful                                                                             
  [i] Received 57322 domains

  [i] Target: https://mirror1.malwaredomains.com/files/justdomains
  [✓] Status: Retrieval successful                                                                             
  [i] Received 26853 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  [✓] Status: Retrieval successful                                                                             
  [i] Received 34 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [i] Status: Pending...

p928s1984 · June 25, 2020, 1:10pm

After adding back the 4. it no longer will start. its stuck trying to update that one list.

yubiuser · June 25, 2020, 1:13pm

I can open it without problems.

Also pihole -g workes

nanopi@nanopi:~$ pihole -g
  [i] Neutrino emissions detected...
  [✓] Pulling blocklist source list into range

  [✓] Preparing new gravity database
  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  [✓] Status: No changes detected
  [i] Received 34 domains

  [i] Target: https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  [✓] Status: Retrieval successful
  [i] Received 2701 domains

But both lists are deprecated (outdated) and won't be included from pihole v5.1.

github.com/pi-hole/pi-hole

Remove deprecated Disconnect lists from "defaults"

pi-hole:development ← pi-hole:its-getting-awful-quiet-in-here

opened 08:14PM - 26 May 20 UTC

PromoFaux

+1 -5

**By submitting this pull request, I confirm the following:** - [x] I have r…ead and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md), as well as this entire template. - [x] I have made only one major change in my proposed changes. - [ ] I have commented my proposed changes within the code. - [x] I have tested my proposed changes, and have included unit tests where possible. - [x] I am willing to help maintain this change if there are issues with it later. - [x] I give this submission freely and claim no ownership. - [x] It is compatible with the [EUPL 1.2 license](https://opensource.org/licenses/EUPL-1.1) - [x] I have squashed any insignificant commits. ([`git rebase`](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html)) --- **What does this PR aim to accomplish?:** See #3450 for details Removing these lists only removes 337 unique domains from a fresh install with all the lists selected. No brainer really, especially as they're deprecated! With: ``` [i] Number of gravity domains: 87259 (84818 unique domains) ``` Without: ``` [i] Number of gravity domains: 84524 (84481 unique domains) ```

github.com/pi-hole/pi-hole

Disconnect.me list is deprecated

opened 06:55PM - 26 May 20 UTC

closed 03:24PM - 12 Aug 21 UTC

thompiler

Fixed in next release

**In raising this issue, I confirm the following:** `{please fill the checkboxes…, e.g: [X]}` - [x] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [x] The issue I am reporting can be *replicated*. - [x] The issue I am reporting isn't a duplicate (see [FAQs](https://github.com/pi-hole/pi-hole/wiki/FAQs), [closed issues](https://github.com/pi-hole/pi-hole/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), and [open issues](https://github.com/pi-hole/pi-hole/issues)). **How familiar are you with the the source code relevant to this issue?:** (1) I am not personally familiar with the source code related to this behavior other than this appears to be the line where the deprecated file is included: https://github.com/pi-hole/pi-hole/blob/4d25f695267590b61a4061f9bb43448005d99b85/automated%20install/basic-install.sh#L1233 --- **Expected behaviour:** Deprecated lists are not included by default **Actual behaviour:** Deprecated list is included by default **Steps to reproduce:** Run basic install and observe deprecated tracker list is used (`disconnect.me/simple-tracking.txt`). **Debug token provided by [uploading `pihole -d` log](https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#debug):** Not applicable **Troubleshooting undertaken, and/or other relevant information:** Hi there, Full disclosure: I work at Airbrake (https://airbrake.io/). We noticed that this project is using a tracker list file (https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt) that lists Airbrake as an ad tracker. Airbrake is an application monitoring tool (error monitoring, deploy tracking, performance monitoring). Error monitoring is an essential tool for app developers to fix bugs that pop up in their code. It doesn't track users or take screenshots of users' actions. We contacted Disconnect.me (support@disconnect.me) and they shared this with us: > The list you referenced was used in our legacy products. It is not maintained, has not been updated, and is not actively distributed us. Airbrake is not included on our main tracker list (linked below) which is used by partners such as Firefox. > > https://github.com/disconnectme/disconnect-tracking-protection/blob/master/services.json

p928s1984 · June 25, 2020, 1:15pm

I am able to open it with my web browser. very odd.

at least i have a solution!

what is the best place to get a list of all of the "default" "current" "active" "suggested" lists?

yubiuser · June 25, 2020, 1:17pm

Great!

github.com

pi-hole/pi-hole/blob/3aaf9d89b3200d972d06dbb4722953853b0d5617/automated install/basic-install.sh#L1287


      
              do
                  appendToListsFile "${choice}"
              done
              touch "${adlistFile}"
              chmod 644 "${adlistFile}"
          }
          
          # Accept a string parameter, it must be one of the default lists
          # This function allow to not duplicate code in chooseBlocklists and
          # in installDefaultBlocklists
          appendToListsFile() {
              case $1 in
                  StevenBlack  )  echo "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" >> "${adlistFile}";;
                  MalwareDom   )  echo "https://mirror1.malwaredomains.com/files/justdomains" >> "${adlistFile}";;
              esac
          }
          
          # Used only in unattended setup
          # If there is already the adListFile, we keep it, else we create it using all default lists
          installDefaultBlocklists() {
              # In unattended setup, could be useful to use userdefined blocklist.