This becomes extremely important, infact, when one has the installation in a data center. Even if the installation is at the home network, then the detection of a client connecting to pi-Hole services - of any kind - is very helpful for several reasons.
Firstly, one could allow only specific client to connect to a particular port, where the admin interface could be made available. Other clients have no access to that port.
Secondly, if client detection is in place, then one could restrict use of normal pi-Hole services. So only some clients - configured beforehand could access port 53.
Thereafter, one has an added security and could be installed in a data center without much thoughts.
That is what I am working on now and your charming input is - like always - very welcome.
You already know my point of view here: Close any Pi-hole related ports and establish a VPN connection to your datacenter server. You may safely allow any ports to go through the VPN as you can control who is able to connect to the VPN.
In contrast to this, you are asking to e.g limit port 53 access to IP 81.82.83.84 which may be your home IPv4 address. However, all traffic will be unencrypted. Furthermore, it is possible, as you know, to fake the sender's IP address which would still allow abusing your DNS server*. This could, however, easily be mitigated by installing a VPN server which has only been done once and will give you added security afterwards (like for all time).
Having said all that, I'm even unsure if you can actually filter UDP packets by source IP using e.g. IPTables (see *: the sender's IP address is not part of the protocol header).
*) Background information: DNS packets are sent over UDP protocol. The UDP packet itself, however, does not include your (the sender's) IP address at all, but the source IP address is part of the packet it is sent in (in contrast to TCP where it is contained in the header). Hence, it is quite easy to write there whatever I like.
ENCRYPTION IS NOT - ALWAYS - A MUST. Use DNSCrypt.
The TCP traffic encryption/decryption requires resources. Its use should be justified. For watching "normal" videos, I require no encryption. In fact, I would love to make people busy, just like I love to make hackers busy to crack my servers, to make some captures of those video transmissions. Thus, to have all streaming traffic over VPN would be - for me - an exaggeration.
The UDP traffic with DNSCrypt could do the job of circumventing some eyes in the middle, if one requires them to remain closed.
SHIFTING THE OUTBOUND POINT OF PRESENCE DOES NOT ALWAYS HELP
After new technologies developed, "normal users" became unaware, vulnerable and helpless. Earlier, it helped to shift the IP of an outbound connection. This is no longer valid. Today, one has to be really extremely clever to understand many elements of an outbound connection to be able to control it.
For e.g., I am able to "more-or-less" able to "turn-off" the Google dynamic tracking while typing. I do not let alphabets in the search box to directly flow into Google's servers and handle precisely what goes out. Often, I have a Google search page generated where I could make a direct link "which circumvents" any further registration of clicks. Most of you guys would not even notice of such technologies, leave apart imposing controls.
By implementing your idea of "VPN", I loose some aspects of my control because the outbound-point-of-presence gets shifted on TCP protocol, i.e. from my workstation to that of my pi-Hole server.
HAVE LOCAL TCP TRAFFIC UNDER FINE GRAINED CONTROL
By having the outbound-point-of-presence in my workstation, I can apply a fine grained control based on DNS-IP mapping per application.
The control of outbound TCP traffic gets lost or reduced with VPN because everything gets concentrated to one channel.
THE MYTH: "FAKE THE SENDER'S IP ADDRESS" DOES NOT SCARE ME!
While advocating this myth "fake the sender's IP address", I would very much like to invite you to hack my server. If and as you advocate this, you may also know this. Or did you just read this stuff somewhere?
To be able to "fake the sender's IP address", one requires a lot of expertise and technological resources. The Governments and security agencies or some engineers may have these capacities. If one has, then I would ask a question, why would one want to do that and for what?
ADVANTAGES OF HAVING A HOST-BASED NETWORK ACCESS CONTROL
The advantages of faking the sender's IP for the purpose are useless for these people.
The advantages of having a host-based Network Access Control are many. Some are listed here:
USE TCP WRAPPERS AND XINETD FOR REMOTE TCP POINT
By using Xinetd, one could add more security. By having a DynHost IP added to a subdomain and have that poured into the server changes the IP that could connect to it on a daily basis. Thus, I conclude - or concluded - that there is no real risk to have pi-Hole installed in a datacenter. Htaccess helps here too but this could be restricted to /admin/.
If my routers IP changes and if the dynamic IP is mapped in the config of my server, it becomes extremely difficult for someone to identify the router's IP.
One could use xinetd libwrap to identify the dynamic IP of the router of the home network. This rules out any misuse of port 53.
NETWORK ACCESS CONTROL could be helpful in Home network
Thus, if there is NETWORK ACCESS CONTROL implemented in the scripts, then one could even apply some more controls and features, for e.g. who could see an ads blacklisted page, or if one has seen it then if he could have an access to login to have it white listed.