Testing A YaCy Search Engine with High DNS Loading and invalid domains

[smokingwheels]

I have things like these turning up in the Query logs and are now on my blocklists...

I will let the Yacy developers know also.

AAAA dpird.wa.gov.au?subject= OK (cache) NXDOMAIN (0.4ms)
AAAA dpird.wa.gov.au?subject= OK (cache) NXDOMAIN (1.2ms)
A dpird.wa.gov.au?subject= OK (answered by dns.quad9.net#53) NXDOMAIN (1.2ms)
A dpird.wa.gov.au?subject= OK (sent to dns.quad9.net#53) N/A

customer_name.atlassian.net"	
charts.gitlab.io<	
www.mumble.info">mumble<	
ayearofsprings.crd.co">a	
maven.google.com"	
toasters.example",	
www.apple.com"	
apptwo.company.com"	
www.yosoygames.com.ar%5c	
www.example-dsp.com',	
www.example-ssp.com',	
buyer2.com',	
www.example-dsp.com'	
www.another-buyer.com'	
www.some-other-ssp.com',	
risc.example.com",

I am able to block them if I catch them, but can't add them on the Domain interface page.

I use the teleporter to share the lists between Piholes which is good and helps YaCy stop crashing so often.

This Pihole pic is while an initial Crawling of one site at a depth of 6 starts up with crawl rate of 30 000 set in crawler.

Note: This is only Half the DNS requests because I use a dnsmasq server to load share the between two Piholes.

Thank you all, for Creating, Sharing and Updating the Pihole Project.

[chrislph]

@smokingwheels

I have things like these turning up in the Query logs

It looks like YaCy is allowing malformed DNS requests to be made (for non-domains based on incorrectly parsed strings), and since you are using Pi-hole for DNS these requests are visible to you in Pi-hole's log. Good idea to let their devs know.

@smokingwheels

I am able to block them if I catch them, but can't add them on the Domain interface page.

You can only add valid domains to the Domains page. I wondered if it might be possible to do it from the terminal, but no, it is rightfully rejected.

$ pihole -b apptwo.company.com\"
[✗] apptwo.company.com" is not a valid argument or domain name!

I'm not sure what the effect might be from adding malformed domains via the Query Log as you catch them. It might be best to not do that (thinking unintentional sideffects from non-domain characters in domainlist in Gravity?) – or at least keep track of them and delete them as soon as YaCy's behaviour is fixed.

[smokingwheels]

@chrislph

I'm not sure what the effect might be from adding malformed domains via the Query Log as you catch them.

A positive effect I can crawl 1 level deeper on a site with yacy crashing less often.