The Ingress API is a good example of the API standardization that Kubernetes offers. Many cloud-native components, such as ExternalDNS, Traefik and cert-manager, integrate with the Ingress API, leading to a consistent experience.
Over time, the limitations of the Ingress API have led to the creation of various ad-hoc CRDs that aim at offering a better abstraction. Istio’s VirtualService CRD is one such example, and each proxy or service mesh creates its own.
This represents a challenge for tools such as cert-manager and ExternalDNS, that may not be able to support all of these different CRDs. That is where SIG Network Gateway API comes in. The Gateway API project is managed by the SIG Network community and aims to provide a universal API for modelling service networking in Kubernetes.
While the Gateway API is still in its alpha stages, it is already gaining wide adoption. Today, you can use the gateway API with Ambassador, Contour, Gloo, HAProxy, Istio, Kong and Traefik.
cert-manager in v1.5 added experimental support for the Gateway API. cert-manager can solve HTTP-01 challenges using HTTPRoute resources, along with the possibility to have Certificate resources automatically created by using annotations on Gateway resources. The documentation for using the Gateway API with cert-manager is available on the page Securing Gateway Resources.
This blog post will walk you through the process of using cert-manager with ExternalDNS and Traefik using the Gateway API.
A Service type=LoadBalancer controller installed (if you are running on GKE, then you are good to go).
A DNS zone in some DNS provider that ExternalDNS supports (e.g., CloudDNS or Scaleway). You can also use the IP addresses directly, but this guide will focus on host names instead.
In the remainder of this guide, we will be using:
a pre-existing GKE cluster,
a pre-existing CloudDNS zone (domain mael-valais.example.com.).
Before installing the Gateway API implementation, we will need to install ExternalDNS.
Note that as of August 6, 2021, ExternalDNS does not support the Gateway API natively. Andy Bursavich is working on implementing it and you can follow the development on the issue external-dns#2045.
Let us set a couple of variables:
ExternalDNS needs a service account key to let ExternalDNS configure DNS records on the zone:
Finally, we can install ExternalDNS:
Let us configure the Service type=LoadBalancer created by Traefik with a DNS name:
After some time, you should see:
As detailed on the page Traefik & Kubernetes, Traefik needs some extra RBAC rules:
Let us install the Gateway API CRDs:
The next step is to install cert-manager. The Gateway API is supported since v1.5.0:
Now, we can create an ACME Issuer and two Gateways: one for solving HTTP-01 challenges and one for listening on 443
And finally, let us create a Deployment to test that out:
You should be able to access the service with the following command:
Known bugs and limitations in Gateway API implementations
While testing this feature with various Gateway API implementations we found some issues that will cause problems in some circumstances.
[HAProxy Ingress] certificateRef.group cannot be set to core
Update October 5, 2021: this issue was fixed in HAProxy-ingress v0.13.4 (see PR).
As of haproxy-ingress v0.13.0-beta.2, haproxy-ingress expects certificateRef.group to be empty, but the Gateway API v1alpha1 CRD requires a non-empty group (issue haproxy-ingress#830).
This issue has been fixed for v1alpha2 in PR 562, but will not be back-ported to v1alpha1.
Until PR 833 is merged, the only workaround I know about is to manually disable the non-empty requirement from the CRD:
[Traefik] HTTPRoute and Gateway must be on the same namespace
Update October 5, 2021: HTTPRoute and Gateway on different namespaces are now supported as of Traefik 1.8.0 (see PR).
As of Traefik 2.4.9, Traefik only watches for HTTPRoutes that are on the same namespace as the Gateway and does not honor from: All (issue traefik#8246).
For example, the following won’t work:
At this point, you would expect to be able to create a Certificate in any namespace with an ACME Issuer. Let’s imagine that you have an ACME Issuer and Certificate in the namespace default:
cert-manager will create, as expected, an HTTPRoute in the namespace default:
But Traefik won’t do anything with it:
[Traefik] One faulty listener breaks the entire Gateway
Traefik requires all listeners to be valid before configuring itself, which means we can’t use a single Gateway for both the HTTP-01 challenges on port 80 and 443 configured with a certificate that is meant to be created using the HTTP-01 challenge.
For example:
Even thought the first listener is valid, none of them will be configured in Traefik; Traefik shows the following error:
An error occurred while creating gateway status: 1 error occurred: Error while retrieving certificate:
secret default/traefik-tls does not exist
Imagining that the Secret traefik-tls existed, cert-manager would remove the temporary HTTPRoute that it created, which means the first listener (on port 80) would start erroring, preventing the second listener (on port 443) from being configured:
time="2021-08-05T16:28:32Z" level=error msg="an error occurred while creating gateway status: 1 error occurred:
Cannot fetch HTTPRoutes for namespace "default" and matchLabels map[gateway:http01-solver]
The only workaround is to create a separate Gateway to prevent one listener from crashing the other listeners:
[ExternalDNS] hostnames in Gateway and HTTPRoutes are not supported
One great feature of ExternalDNS is that it picks up the hostnames on your Ingress resources and creates A records for them. Unfortunately, ExternalDNS does not support the Gateway API.
A conversation is currently ongoing in external-dns#2045 to add support for the Gateway API.
A work around used in this guide was to set the external-dns.alpha.kubernetes.io/hostname annotation:
Closing remarks
The Gateway API, although still in early stage, brings a set of features that better abstract how gateway proxies are operated nowadays. Like the Ingress API, we can foresee that it will become a de facto standard, with many cloud-native components supporting it.
With this experimental support in cert-manager we are helping the project to grow, and finding issues that can be fixed to improve the experience for users.
As demonstrated here you can try out the Gateway API today, provide feedback to the project about whether it works well for you, and help guide it to production readiness.
