It would be nice to have an option to change the aggregation of the Top Blocked Domains / Top Permitted Domains to ignore the sub domain component.
For example:
- mask.icloud.com 1327
- mask-h2.icloud.com 1298
- metrics.icloud.com 739
Would just be aggregated as
- icloud.com 3364
The actual breakdown of the sub domain count could be shown with an on hover window when mousing over the aggregated domain or just have click drill through if it's easier to implement.
In my opinion while less specific this would provide a more useful view of which services are actually being blocked allowed etc. Currently 6 of my top 10 allowed entries are only two actual domains and my top 10 blocked all 10 are only three actual domains.
Just a note:
I understand your request, but there is no "breakdown" happening in the web interface.
From DNS queries point of view there is no relation between a domain and subdomains. They are different domains and these queries are individually stored in the database.
example.com is a domain.
www.example.com is another domain.
pi-hole.example.com is another domain.
dns.pi-hole.example.com is another domain.
That's interesting. I didn't know that.
I assume there are libraries to parse a URL into it's component pieces. I wonder if that's a viable option. Given the absurd complexity of some domains in the current day it just might not be worth even trying.
There are functions to parse the domain parts, but they are not related to your request.
I think this example could make it clear:
When a DNS query is made for www.example.com, no queries are made (or stored) to example.com or com. Just one query to www.example.com.
In the database, there are no relation between this domain and "other domain levels".
No, I get what you’re saying, totally understand.
What I’m saying is either a column could be added to the database so you have actual_domain (www.example.com) and primary domain (example.com) for each row.
When querying the DB to get the top 10 you perform do it at query time using a DB function or regex (I’m not sure what SQLite supports).
The former would be much more performant. The latter easier to implement.
No, Pi-hole will definitely show ten distinct domains.
You are suggesting to aggregate DNS request counts by their second level domain parts.
I see very little benefit in this, with a considerable potential to make statistics much less significant or even meaningless.
As there are no inherent semantics attached to a domain beyond what the DNS hierarchy itself would imply, a second level aggregation would be a rather arbitrary criterion.
It may already fail to even meet your expectation for those *.icloud.com domains specifically, as aggregation may not be limited to those three domains you quote in your example.
There are literally hundreds of other domains that end in icloud.com (click for a sample excerpt)
ic1-wopi.icloud.com
ic2-wopi.icloud.com
ic3-wopi.icloud.com
ic4-wopi.icloud.com
icloud4-hubble.icloud.com
mailws.icloud.com
mr11p00im-monitorm001.monitorm.icloud.com
mr11p00im-monitorm002.monitorm.icloud.com
mr11p00im-monitorm003.monitorm.icloud.com
mr11p00im-monitorm004.monitorm.icloud.com
mr11p00im-monitorm005.monitorm.icloud.com
mr11p00im-monitorm006.monitorm.icloud.com
mr11p24im-monitorm001.monitorm.icloud.com
mr11p24im-monitorm002.monitorm.icloud.com
mr11p24im-monitorm003.monitorm.icloud.com
mr11p26im-monitorm001.monitorm.icloud.com
mr30124001c.connectivity.icloud.com
mr30124002c.connectivity.icloud.com
mr30124003c.connectivity.icloud.com
mr30124004c.connectivity.icloud.com
mr30126001c.connectivity.icloud.com
mr30126002c.connectivity.icloud.com
mr30126003c.connectivity.icloud.com
mr30126004c.connectivity.icloud.com
mr30128001c.connectivity.icloud.com
mr30128002c.connectivity.icloud.com
mr-e3sh.icloud.com
mx001.icloud.com
mx01.mail.icloud.com
mx02.mail.icloud.com
mx1.mail.icloud.com
mx2.mail.icloud.com
mx3.mail.icloud.com
mx4.mail.icloud.com
mx5.mail.icloud.com
mx6.mail.icloud.com
newspublisherapi-old.icloud.com
nk11p00mm-monitorm001.monitorm.icloud.com
nk11p00mm-monitorm002.monitorm.icloud.com
nk11p00mm-monitorm003.monitorm.icloud.com
nk11p00mm-monitorm004.monitorm.icloud.com
nk11p00mm-monitorm005.monitorm.icloud.com
nk11p00mm-monitorm007.monitorm.icloud.com
nk11p00mm-monitorm008.monitorm.icloud.com
nk11p00mm-monitorm009.monitorm.icloud.com
nk11p00mm-monitorm011.monitorm.icloud.com
nk11p03mm-monitorm001.monitorm.icloud.com
nk20001001.connectivity.icloud.com
nk20001002.connectivity.icloud.com
nk20001003.connectivity.icloud.com
nk20002001.connectivity.icloud.com
nk20002002.connectivity.icloud.com
nk20002003.connectivity.icloud.com
nk20003001.connectivity.icloud.com
nk20003002.connectivity.icloud.com
nk20003003.connectivity.icloud.com
nk20004001.connectivity.icloud.com
p01-btmmconduit.connectivity.fg1ad.icloud.com
p01-btmmconduit.connectivity.mj2kh.icloud.com
p01-btmmconduit.connectivity.qr3no.icloud.com
p01-btmmconduit.connectivity.xs4tz.icloud.com
p01-conduit.connectivity-old.icloud.com
p01-mailws-old.icloud.com
p02-btmmconduit.connectivity.fg1ad.icloud.com
p02-btmmconduit.connectivity.mj2kh.icloud.com
p02-btmmconduit.connectivity.qr3no.icloud.com
p02-btmmconduit.connectivity.xs4tz.icloud.com
p02-conduit.connectivity-old.icloud.com
p02-mailws-old.icloud.com
p03-btmmconduit.connectivity.qr3no.icloud.com
p03-conduit.connectivity-old.icloud.com
p03-mailws-old.icloud.com
p04-btmmconduit.connectivity.qr3no.icloud.com
p04-conduit.connectivity-old.icloud.com
p04-mailws-old.icloud.com
p05-conduit.connectivity-old.icloud.com
p05-mailws-old.icloud.com
p06-mailws.icloud.com
p07-conduit.connectivity-old.icloud.com
p07-mailws-old.icloud.com
p08-mailws.icloud.com
p08-mailws-old.icloud.com
p09-conduit.connectivity.icloud.com
p09-mailws-old.icloud.com
p10-conduit.connectivity.icloud.com
p10-imap.mail.icloud.com
p10-mailws-old.icloud.com
p11-conduit.connectivity.icloud.com
p12-conduit.connectivity.icloud.com
p13-conduit.connectivity.icloud.com
p14-conduit.connectivity.icloud.com
p15-conduit.connectivity.icloud.com
p16-conduit.connectivity.icloud.com
p17-conduit.connectivity.icloud.com
p18-conduit.connectivity.icloud.com
p19-conduit.connectivity.icloud.com
p203-imap.mail-china-old.icloud.com
p203-mailws.icloud.com
p203-smtp.mail-china-old.icloud.com
p204-imap.mail-china-old.icloud.com
p204-mailws.icloud.com
p204-smtp.mail-china-old.icloud.com
p25-btmmconduit.connectivity.icloud.com
p26-btmmconduit.connectivity.icloud.com
p27-btmmconduit.connectivity.icloud.com
p28-btmmconduit.connectivity.icloud.com
p28-conduit.connectivity.icloud.com
p29-conduit.connectivity.icloud.com
p30-conduit.connectivity.icloud.com
p31-conduit.connectivity.icloud.com
p32-conduit.connectivity.icloud.com
p33-conduit.connectivity.icloud.com
p34-conduit.connectivity.icloud.com
p35-conduit.connectivity.icloud.com
p36-conduit.connectivity.icloud.com
p73-smtp.mail.icloud.com
p74-smtp.mail.icloud.com
p75-smtp.mail.icloud.com
p76-smtp.mail.icloud.com
p77-smtp.mail.icloud.com
p78-smtp.mail.icloud.com
p79-smtp.mail.icloud.com
p80-smtp.mail.icloud.com
p97-iworkpreviewapi-old.icloud.com
p98-iworkextstore-old.icloud.com
p98-iworkpreviewapi-old.icloud.com
p99-mailws.icloud.com
pv40133001c.connectivity.icloud.com
pv40133002c.connectivity.icloud.com
pv40133003c.connectivity.icloud.com
pv40133004c.connectivity.icloud.com
pv40136001c.connectivity.icloud.com
pv40136002c.connectivity.icloud.com
pv40136003c.connectivity.icloud.com
pv40136004c.connectivity.icloud.com
pv40138001c.connectivity.icloud.com
pv40138002c.connectivity.icloud.com
pv-e3-origin.icloud.com
pv-e3sh.icloud.com
pvp00-e3.icloud.com
s00im-imap.mail.icloud.com
s00m-smtp.mail.icloud.com
s01-ckdatabase-old.icloud.com
s01-ckdevice.icloud.com
s01-ckshare.icloud.com
s01-content-old.icloud.com
s01-documentvip001.icloud.com
s01-escrowproxy.icloud.com
s01im-imap.mail.icloud.com
s01im-smtp.mail.icloud.com
s01-keyvalueservice001.icloud.com
s01-setup.icloud.com
s02-ckdatabase-old.icloud.com
s02-ckdevice.icloud.com
s02-ckshare.icloud.com
s02-content-old.icloud.com
s02-documentvip001.icloud.com
s02-escrowproxy.icloud.com
s02im-imap.mail.icloud.com
s02im-smtp.mail.icloud.com
s02-keyvalueservice002.icloud.com
s02-setup.icloud.com
smtp.icloud.com
smtp.mail-old.icloud.com
st10001001.connectivity.icloud.com
st10001002.connectivity.icloud.com
st10001003.connectivity.icloud.com
st10002004.connectivity.icloud.com
st10002005.connectivity.icloud.com
st10002006.connectivity.icloud.com
st10003007.connectivity.icloud.com
st10003008.connectivity.icloud.com
st10003009.connectivity.icloud.com
st10004010.connectivity.icloud.com
st11b01mm-monitorm201.monitorm.icloud.com
st11b01mm-monitorm202.monitorm.icloud.com
st11b01mm-monitorm203.monitorm.icloud.com
st11b01mm-monitorm204.monitorm.icloud.com
st11b01mm-monitorm205.monitorm.icloud.com
st11p00mm-monitorm001.monitorm.icloud.com
st11p00mm-monitorm002.monitorm.icloud.com
st11p00mm-monitorm003.monitorm.icloud.com
st11p00mm-monitorm004.monitorm.icloud.com
st11p00mm-monitorm005.monitorm.icloud.com
Furthermore, it may produce unexpected or unwanted results for other domains as well, e.g. your suggestion would also aggregate all of *.co.uk domains as well as *.in-addr.arpa, which in turn may catapult otherwise unremarkable counts into the top 10s.
Obviously, universally applying a second level aggregation would potentially result in a much distorted view of statistics.
Given above considerations, I'm not convinced that implementing this would be beneficial.
Also, as picking an appropriate aggregation level would very much depend on the actual domain, and quite probably on personal preferences as well, that would require quite a bit more customisation than your suggestion anticipated.
1 Like
Yes, I am suggesting second level domain aggregation. I am aware that this would provide significantly less detail than the entire domain.
The point, in my opinion would be to know "who's house" most of your permitted / blocked traffic is going to, not which specific rooms in that house. I'm also only suggesting it as an option rather than a primary change and I'm not suggesting even then, dropping the entire domain counts but rather putting them as a hover window or drill through link on the second level domain.
As an example about 30% of blocked traffic in my network currently is going to logs.netflix.com ... I don't have a netflix subscription. Being able to see all traffic going to *.netflix.com would be great. If these were spread across a number of sub domains then the counts would likely not be enough to appear in the top 10 and therefore I likely wouldn't have become aware of it,
In relation to the domain aggregation being complex, I agree and acknowledged this in my second post. That said, there are a finite number of TLDs, surely functions exist that can find the TLD in a domain and then return TLD plus the next block to the left as the second level domain.