Very slow DNS resolution through Pi-Hole using Stubby

With FTLDNS, directly queried on my Pi 3B:

$ dig yahoo.jp

; <<>> DiG 9.9.5-9+deb8u15-Raspbian <<>> yahoo.jp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36674
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;yahoo.jp.                      IN      A

;; ANSWER SECTION:
yahoo.jp.               3600    IN      A       183.79.23.196
yahoo.jp.               3600    IN      A       183.79.227.111

;; AUTHORITY SECTION:
yahoo.jp.               3600    IN      NS      ns01.yahoo.co.jp.
yahoo.jp.               3600    IN      NS      ns11.yahoo.co.jp.
yahoo.jp.               3600    IN      NS      ns12.yahoo.co.jp.
yahoo.jp.               3600    IN      NS      ns02.yahoo.co.jp.

;; Query time: 842 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Aug 17 23:11:54 CEST 2018
;; MSG SIZE  rcvd: 154

(this was the first round with my local unbound resolver not being aware of this domain)

$ dig yahoo.jp

; <<>> DiG 9.9.5-9+deb8u15-Raspbian <<>> yahoo.jp
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29200
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;yahoo.jp.                      IN      A

;; ANSWER SECTION:
yahoo.jp.               3545    IN      A       183.79.227.111
yahoo.jp.               3545    IN      A       183.79.23.196

;; Query time: 11 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Aug 17 23:12:49 CEST 2018
;; MSG SIZE  rcvd: 69

on my second try. I repeated some more and see reply times between 1 to 12 milliseconds but not more.