Thank you! Your hints helped. It was actually easy to figure out.
If anyone's looking for the solution, here's the URL you need to block (if you are in Europe, it seems):
vsp-alexa-eu.amazon.com
With that one on the blacklist, there's no connection for Alexa to the server.