My home setup is a full-blown Kubernetes cluster (Overkill some might say, but it’s what I do, so it makes sense to have one). As part of this, I have load balancing, provided by MetalLB, and routing to my applications thanks to Nginx Ingress. This is all well and good, and setting up public sites and private sites behind my router, while challenging, has been largely successful. Except for one particularly challenging bit of configuration that, for whatever reason, escaped me until just recently.
Some of the services I run on my cluster are Drone CI (For CI/CD), and a dedicated Docker Registry (thanks to Harbor). For Drone in particular, the configuration is slightly more complex than a normal service. There is an application server, which is fronted by a private-only ingress, and there is a portion of the application that responds to webhooks from GitHub to trigger events, which must be exposed via a public ingress. This means the application has both public _and_ private ingresses, and that anything inside the cluster needs to route its traffic to the private side. The trouble here is when you mix in LetsEncrypt (via Cert Manager). The certificate generated needs to be tied to your public ingress, not your private one, or the HTTP01 validation won’t work, but that’s normal and expected. To do that we can specify the cert-manager.io/cluster-issuer on the ingress that is public, and share the secret between the two. Problem is: Cert Manager needs to validate the temporary ingress it creates as part of its “propagation check”. This will check that DNS resolution is working and the HTTP01 validation will work. However, the hostname that will be resolved here is the same one on the public side as the private side, and inside the cluster it will always resolve to the IP of the internal ingress controller, not the public ingress controller hosting the validation ingress.
One fix to this (and one I used for a while to get by) is to set the DNS resolver to route to the public IP while Cert Manager does its thing. This works, but it then prevents everything on the inside network from reaching the private service at the same time. Only one thing can work at a time in this scenario (And for the last year or so, every three months I’d swap it over, renew my certs, and swap it back). The real fix is to update _only the Cert Manager’s pod_ to resolve to the public IP, and permitting everything else internally to resolve as normal.
I achieved this with the following config:
hostAliases:
- hostnames:
- drone.codethat.rocks
- registry.codethat.rocks
ip: 10.0.10.2
This leverages the Kubernetes built-in functionality to effectively peg the /etc/hosts file in the pod with some overridden DNS resolution, while allowing all other DNS lookups to work as expected.
It makes me sad it took this long to finally come to a working solution, but hopefully this advice can help someone else looking to solve a similar problem. I know I couldn’t find anything out there.
