Hi there, I hope you're doing well.
I have a pihole instance, I get 3600 query per second at max, I use godnsbenchmark to benchmark the pihole instance, I want it to give me at least 100K per second, I gave it 32 GB ram, and 64 core cpu so no limits on hardware side, can you please tell me how can I solve it ?
Please elaborate on your use case and provide more details.
Why do you want/need 100K queries per second?
Thanks for your reply.
Well, here is the story:
I want to use Pihole in a local ISP, our current load is about 300K request per second on our current self developed DNS servers
I want to use Pihole as the frontend and use my dns servers as its Upstream ...
here is the Pihole VM hardware specs:
RAM 64 GB
CPU 32 Core
Storage 150 NVME
NIC 10G
it's on docker now ( I tested it on the linux itself too, the performance is the same)
here is my benchmark tool:
godnsbench -a 127.0.0.1 -p 50000 -q example.com -c 1000000
(GitHub - ameshkov/godnsbench: Simple DNS bench util that supports encrypted protocols.)
You can test it on your pihole and see the results, it would be under 5k per second I guess
I set DNSMASQ MAX concurrent to 100K even
I set shared memeory to 8G, though even it would use about 10MB at most so no problem for that too
I disabled query log, long time DB and everythin that I thought it might be a bottleneck.
what ever you need tell me, I will share
Thanks in advance
Can you run the same benchmark test with a vanilla dnsmasq install and see if the results are the same? That will let us narrow down where to search for the issue by letting us know if the situation is in Pi-hole or in dnsmasq.
What is the exact configuration you are running. I don't see a max concurrent flag for dnsmasq in the man page.
Sorry I mean dns-forward-max=509600, I added this line after pihole restarted multiple times after reaching more than 250 concurrent requests because of DNS_FORWARD_MAX warning, I added this
Here is the configs :
setupVars.conf
INSTALL_WEB_INTERFACE=true
WEBPASSWORD=xx
PIHOLE_INTERFACE=eth0
QUERY_LOGGING=true
BLOCKING_ENABLED=false
WEBUIBOXEDLAYOUT=boxed
WEBTHEME=default-dark
DNSMASQ_LISTENING=single
DNS_FQDN_REQUIRED=true
DNS_BOGUS_PRIV=true
DNSSEC=false
REV_SERVER=false
PIHOLE_DNS_1=172.18.0.21#5353
FTL:
#; Pi-hole FTL config file
#; Comments should start with #; to avoid issues with PHP and bash reading this file
MACVENDORDB=/macvendor.db
LOCAL_IPV4=0.0.0.0
RATE_LIMIT=0/0
and DNSMASQ:
dns-forward-max=509600
addn-hosts=/etc/pihole/local.list
addn-hosts=/etc/pihole/custom.list
localise-queries
no-resolv
log-queries
log-facility=/var/log/pihole/pihole.log
log-async
cache-size=10000
server=172.18.0.21#5353
domain-needed
expand-hosts
bogus-priv
interface=eth0
add-subnet=32
here is dnsmasq benchmark
root@maj0r:/tmp# dnsperf -s 172.18.0.22 -p 53 -c 100 -d queries.txt -n 30
DNS Performance Testing Tool
Version 2.9.0
[Status] Command line: dnsperf -s 172.18.0.22 -p 53 -c 100 -d queries.txt -n 30
[Status] Sending queries (to 172.18.0.22:53)
[Status] Started at: Sun Nov 24 06:57:09 2024
[Status] Stopping after 30 runs through file
[Status] Testing complete (end of file)
Statistics:
Queries sent: 269280
Queries completed: 269280 (100.00%)
Queries lost: 0 (0.00%)
Response codes: NOERROR 269280 (100.00%)
Average packet size: request 27, response 43
Run time (s): 8.175174
Queries per second: 32938.748460
Average Latency (s): 0.002792 (min 0.000082, max 0.004656)
Latency StdDev (s): 0.000163
I'm sure we can reach +50K with some clustering and tuning on dnsmasq
Please do the benchmark on your own instance and you'll get what I'm saying, somehow pihole is limiting the Query per seconds
Even if you could tell me or release a new version that could handle 50K is enough for me but my final number would be 300K per second, we have enough hardware
Please help me, I'm really stuck
Don't do that. That's the upper limit on how many queries can be 'in flight' at any time and if you have 50k queries waiting to be resolved then you're really screwed.
--dns-forward-max=
queries
Set the maximum number of concurrent DNS queries. The default value is 150, which should be fine for most setups. The only known situation where this needs to be increased is when using web-server log file resolvers, which can generate large numbers of concurrent queries. This parameter actually controls the number of concurrent queries per server group, where a server group is the set of server(s) associated with a single domain. So if a domain has it's own server via --server=/example.com/1.2.3.4 and 1.2.3.4 is not responding, but queries for *.example.com cannot go elsewhere, then other queries will not be affected. On configurations with many such server groups and tight resources, this value may need to be reduced.
Are you willing to financially sponsor this request?
It doesn't matter how many cores you offer, all UDP DNS load is done in a single thread in dnsmasq. Only dedicated TCP connections can live in parallel. I'm afraid your use-case may simply be unavailable. We have reports of Pi-hole being deployed at a university-scale network with roughly 100,000 active clients at the same time, yet, this does not even come close to 300k per second.
I guess the only way to offer what you want is parallelism due to spreading your workload across many Pi-holes. Depending on your hardware, they may very well be virtualized, e.g, docker containers, maybe even on the same host.
Thanks, I will use docker swarm or k8s to see how many concurrent queries it can handle, if you have optimized docker-compose file for swarm env please drop it here 
Actually I can't afford money for that, off-course I will donate some to the project but I really can't do big stuff in that matter
I reallllly appreciate if you guys join me in this journey to move the Pi-Hole to upper echelon
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.