Python script to add adlists in bulk

I made a little python3 tool that makes bulk adding very easy. I'm only adding this here because it was the first discussion I found when searching, so others may too. Yes it makes me feel gross and I'm sorry.

2 Likes

jessedp

I have hit a dead end trying to load your python tool

latest pi hole installed on stretch lite

I tried downloading/compiling 3.8.3 using some other guides online...

    sudo apt-get install build-essential checkinstall
    sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev

    sudo wget https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz
    sudo tar -zxvf Python-3.8.3.tgz
    sudo ./configure --enable-optimizations
    sudo make altinstall

python --version shows 3.8.3 but I get a python >=3.5 required error

I tried apt remove other versions I found (python3 and 3.5) and I ended up losing pip3

So I removed and tried 3.8.3 again and still no joy.

Thoughts?

Thanks!

Is this github issue you, too? If so, have you tried following the instructions listed there for building python?

Here's a docker image for stretch-slim that probably has all the commands/packages you need for it.

If that doesn't work, maybe use a different distro if possible? That answer sucks, but having a working python install is kind of a requirement and I can't go through and fix that on every variant of os configuration people may have.

That is not me.

I'm using pi zero hence trying to keep everything lite/small

I understand the reasoning... did you try the steps they used to build that docker image for the stretch-slim?

Not yet.

I will try on a fresh SD flash tomorrow and report back. Interesting there isn't already a cleaner way to update. I removed a python3.5 folder when I started deleting things, I'll see if its there as part of the clean install. Might just be an issue of changing default version or loading missing components.

Thanks!

Ok, fresh install of pihole over stretch-lite

python --version
Python 2.7.13

python3 --version
Python 3.5.3

pip3
-bash: pip3: command not found

directories in /usr/lib/
python2.7
python3
python3.5

I see no 'pip' files in /usr/lib/python3.5 so technically 3.5 is there but incomplete?

Maybe use something like python-virtualenv (like here) or perhaps pyenv (site here)

pyenv seems to work but I still got an error:

 $ sudo pip3 install pihole5-list-tool --upgrade
Collecting pihole5-list-tool
Exception:
Traceback (most recent call last):
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 837, in _validate_conn
    conn.connect()
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:704: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 353, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 313, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.piwheels.org', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python3/dist-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/share/python-wheels/CacheControl-0.11.7-py2.py3-none-any.whl/cachecontrol/adapter.py", line 47, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/share/python-wheels/requests-2.12.4-py2.py3-none-any.whl/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/share/python-wheels/urllib3-1.19.1-py2.py3-none-any.whl/urllib3/util/retry.py", line 315, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Thoughts? I installed 3.8.3 btw, perhaps I will try 3.6.x

so I tried installing 3.6.8 and I get the >=3.6

pi@raspberrypi:~ $ pyenv versions
  system
  3.6.8
* 3.8.3 (set by /home/pi/.pyenv/version)
pi@raspberrypi:~ $ pyenv which pip
/home/pi/.pyenv/versions/3.8.3/bin/pip
pi@raspberrypi:~ $ pyenv global 3.6.8
pi@raspberrypi:~ $ pyenv which pip3
/home/pi/.pyenv/versions/3.6.8/bin/pip3
pi@raspberrypi:~ $ sudo pip3 install pihole5-list-tool --upgrade
Collecting pihole5-list-tool
  Using cached https://www.piwheels.org/simple/pihole5-list-tool/pihole5_list_tool-0.4.5-py2.py3-none-any.whl
pihole5-list-tool requires Python '>=3.6' but the running Python is 3.5.3
pi@raspberrypi:~ $ python --version
Python 3.6.8

Thanks

@sadpanda Can you open an issue on jessedp's GitHub for this? That would be a better place to troubleshoot and would keep this thread clear for discussion on the working script.

Per advice I tired a different disto (DietPi)

loaded PiHole and pip via DietPi package loader

result was

root@DietPi:~# pip list
 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
 Package    Version
 ---------- -------
 pip        20.1.1
 setuptools 44.1.1
 wheel      0.34.2

 root@DietPi:~# python3 --version
 Python 3.7.3

however on run I received error (abbreviated):

ModuleNotFoundError: No module named 'pkg_resources'

with some google-fu this fixed the issue:

sudo pip3 install setuptools==47.1.1

thanks