Thanks, this hint was very helpful (even if it doesn't have anything to do with CNAMEs). I can reproduce this with your suggested test.
edit Content largely replaced/updated
There seems to be an issue with the "fixed" retry algorithm of dnsmasq. I reported this upstream. The issue is that certain queries are refused by the upstream (SERVFAIL). This is by design of the IPv6 test. dnsmasq retries those SERVFAILs while it shouldn't. Further investigation is going on.
I pushed a small fix that should detect and handle this case. Please update on this branch and try again.