This seems reasonable. From the manual it sounds possible that the modem may be hijacking and redirecting all DNS requests.
To test if it is hijacking dns requests, you can check if you are able to get out to cloudflare's DNS, and it should return a code for which you connect to. (Testing by ping isn't effective as it uses ICMP, and if your router or ISP are redirecting UDP port 53, pings will still pass through unscathed).
dig +short CHAOS TXT id.server @1.1.1.1
eg in my case MEL:
rob@builder:~ $ dig +short CHAOS TXT id.server @1.1.1.1
"MEL"
Try this before and after enabling Pi-hole as your DNS. If it doesn't come up with an identifier, then your router is likely interfering with DNS requests.
If this works, then to test for a loop, try the following on the system running Pi-hole:
dig +trace www.google.com @127.0.0.1
If you're needing tricks to bring up the menu, it's likely that the router doesn't fully support the static dns even if you do set it (potentially custom firmware from ISP in place). There is a good chance it is still operating in its default "DNS Proxy" mode, but using your Pi-hole as the destination. All requests from your Pi-hole would then go in a loop.