I think it's that test site. Every time I set up Unbound anywhere, I see this behaviour from that test site. Eg right now on my Unbound, the first attempt times out, the second attempt gets the correct failure answer, no change between attempts from my setup.
$ dig fail01.dnssec.works @127.0.0.1 -p 5335
; <<>> DiG 9.16.48-Debian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; connection timed out; no servers could be reached
$ dig fail01.dnssec.works @127.0.0.1 -p 5335
; <<>> DiG 9.16.48-Debian <<>> fail01.dnssec.works @127.0.0.1 -p 5335
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 38969
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
...