sorry it's not really a problem with behavior but more a question about how does it work.
Does the custom dns server replace completely the one from the list if some are selected?
Because hen I do a nslookup it gives me a serverFAIL when I entered a bad custom IP and also some dns from the list....
No. All upstream DNS servers are used. There is an algorithm within Pi-Hole for determining the best performing upstream server as described in this section of the Pi-Hole documentation:
What are the specific upstream servers you are using and which one is failing? It would be helpful if you generate a debug log, upload it and post the token here. That will let us see your Pi-Hole configuration and performance.