I just fixed it. The issue was that I had bind9 running on my ubuntu box and it was using port 53 without me even knowing it.
So I had 2 DNS servers running at the same time. When I stopped bind9 service
service bind9 stop
Then restarted the pihole, it worked. What made this a huge pita to troubleshoot is that I had no indication that the DNS server wasn't able to access port 53 other than seeing that I couldn't ping inside of the container.
I decided to see what was using port 53 on my host machine and realized then that named (bind9) was running.
I guess if this happens to anyone else, make sure you are only running 1 DNS server. I think that's pretty obvious, but sometimes it's hard to check these things unless you know specifically what you are looking for (ie checking if port 53 is in use).
What made this even harder for me to pin down was that I DID stop bind9 at some point, and my pihole worked temporarily during that time, but I guess bind9 restarted again without me knowing.
I have now removed bind9 completely from my system to prevent that from happening again. Hopefully this helps someone else in the future.
This can be closed!