This post has been updated for Istio version 1.11.4
A service mesh is an architectural pattern that provides common network services as a feature of the infrastructure. This typically includes features such as service discovery and policy enforcement to control how services within the mesh can communicate with each other.
Istio is a service mesh implementation that works by running an instance of Envoy alongside each instance of each of your services to intercept and proxy service traffic. Additionally, fleets of standalone Envoys are deployed to handle traffic entering and leaving the mesh; Istio’s main purpose then is to configure and expose the functionality of Envoy.
In addition to the core features, Istio also supports powerful extension points, as well as the ability to apply custom configuration to the Envoy sidecars. Here we will describe how Istio can be configured to manage the OpenID Connect (OIDC) authentication flow for applications running within the mesh to allow both authentication and authorisation decisions to be offloaded to Istio. There are a number of ways to achieve this with Istio however here we look at two solutions and how their integration points have been affected by changes to Istio’s architecture.
OIDC
OIDC is an identity layer built upon the OAuth 2.0 protocol which allows the identity of a user to be verified based on authentication to an identity provider. There are different types of authentication flow which dictate how authentication is handled by the identity provider, but the most common is the Authorization Code Flow, which we will use here.
Typically, when a user first visits an HTTP service that implements the OIDC Authorization Code Flow, they are redirected to an identity provider (for example Google, Azure or dex) where they would login to obtain an Authorization Code. The user is then redirected again back to the original service, passing the Authorization Code as an HTTP query parameter. The service (or OIDC client) can then exchange this Authorization Code (using its Client ID and Client Secret) for a JWT called an ID Token. This JWT is signed by the identity provider (which the service should verify) with fields (or claims) that contain information about the user who logged in (for example their email address) and can be used to make authentication and authorization decisions. Note that since the user does not have access to the service’s Client Secret, they are unable to obtain such a JWT locally. Instead, in order to prevent the user from having to login on every request, the service can implement login sessions; one way in which this can be achieved is by setting an HTTP cookie and then injecting the obtained JWT into subsequent requests based on the presence of this cookie. Session management is discussed further for our setup below.
OIDC is a common way of delegating the responsibility of managing user credentials to a third-party identity provider and a powerful feature of Istio is that it can be leveraged to manage this flow without your backend service needing to be aware OIDC is even being used.
For more information on OIDC and associated terminology, Okta has a great primer.
Istio 1.4
Istio 1.4 and earlier included a component called Mixer which formed part of the Istio control plane. When policy checks were enabled, before Envoy made an upstream connection it would make a logical request to Mixer in order to determine whether the connection was allowed and what action to take. Mixer therefore provided an extension point for Istio, allowing integrations with external components that could make these policy decisions on its behalf.
The App Identity and Access Adapter is an example of an external component that interfaces with Mixer in exactly this way; by analyzing attributes sent by Envoy to Mixer when making policy decisions it can work out whether a user has already authenticated to a configured identity provider and if not trigger the OIDC flow. The resulting JWT can then be compared against policy configuration to either allow or deny access to the upstream service.
Istio 1.5 and Above
Since Istio 1.5, Mixer has been deprecated in favour of implementing these extensions within Envoy itself. In particular this reduces the latency of policy decisions that would otherwise require a network call to Mixer.
Here we describe in detail an alternative way to configure Istio to manage the OIDC authentication flow and authorization decisions but without Mixer. For the purpose of this description we will assume we are running on a 1.19 GKE cluster with Istio 1.11.4 installed, but this setup should be compatible with recent versions of both Kubernetes and Istio:
gcloud container clusters create istio \
--cluster-version 1.19 \
--machine-type n1-standard-2 \
--enable-autoscaling \
--max-nodes=5
# Install istioctl for Linux or OS X -- if you are on Windows you can download directly from GitHub
# https://github.com/istio/istio/releases
export ISTIO_VERSION=1.11.4
curl -sL https://istio.io/downloadIstioctl | sh -
# Install Istio configured with our custom extension provider to handle the OIDC authentication flow
# https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/#MeshConfig-ExtensionProvider
$HOME/.istioctl/bin/istioctl manifest install -y -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
extensionProviders:
- name: oauth2-proxy
envoyExtAuthzHttp:
service: oauth2-proxy.oauth2-proxy.svc.cluster.local
port: 4180
includeRequestHeadersInCheck:
- cookie
headersToUpstreamOnAllow:
- authorization
headersToDownstreamOnDeny:
- set-cookie
EOF
# Lock down mutual TLS for the entire mesh
# https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/#lock-down-mutual-tls-for-the-entire-mesh
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
EOFWe will deploy Nginx to the cluster to act as the test application we want to configure authenticaiton and authorisation for. You will need to configure a domain (I will use nginx.lukeaddison.co.uk) to expose Nginx on. It should be pointing at the following IP address (this may take a few moments to provision):
kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'We also deploy cert-manager to provision a Let’s Encrypt TLS certificate for us.
WARNING: Once OIDC authentication is enforced on the Istio ingress gateway, cert-manager will no longer be able to renew the certificate using the HTTP-01 challenge solver configured below. One solution would be to use a DNS-01 challenge solver instead
# Set your Nginx domain
NGINX_DOMAIN=""
# Install cert-manager
CERT_MANAGER_VERSION="v1.6.1"
kubectl apply -f "https://github.com/jetstack/cert-manager/releases/download/$CERT_MANAGER_VERSION/cert-manager.yaml"
# Wait for cert-manager to be ready and then configure ingress with TLS termination. Note that we do
# not configure HTTPS redirection to support Let's Encrypt ACME HTTP-01 challenges
kubectl apply -f - <<EOF
# https://istio.io/latest/docs/tasks/traffic-management/ingress/kubernetes-ingress/#specifying-ingressclass
apiVersion: networking.k8s.io/v1beta1
kind: IngressClass
metadata:
name: istio
spec:
controller: istio.io/ingress-controller
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt
solvers:
- http01:
ingress:
class: istio
---
apiVersion: v1
kind: Namespace
metadata:
name: nginx
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: nginx
spec:
selector:
app: nginx
ports:
- name: http
port: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: nginx
namespace: nginx
spec:
hosts:
- $NGINX_DOMAIN
gateways:
- istio-system/nginx
http:
- route:
- destination:
port:
number: 80
host: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: nginx
spec:
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx
namespace: istio-system
spec:
secretName: nginx-tls
issuerRef:
name: letsencrypt
kind: ClusterIssuer
dnsNames:
- $NGINX_DOMAIN
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: nginx
namespace: istio-system
spec:
selector:
app: istio-ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- nginx/$NGINX_DOMAIN
tls:
credentialName: nginx-tls
mode: SIMPLE
EOFAfter a few moments Nginx should become available to the internet which you can verify by curling your domain:
curl "https://${NGINX_DOMAIN}"Authentication
We can now configure OIDC authentication. The first thing we need to do is to configure our OIDC identity provider; we will be using Google here but any compatible provider should work.
For Google specifically, we need to create a Google OAuth application. The Kubeflow documentation walks through the steps to achieve this for IAP. The functional differences for us is that we want to specify Authorized domains to be whatever domain Nginx is exposed on (so nginx.lukeaddison.co.uk for me) and in Authorized redirect URIs specify the /oauth2/callback path of the Nginx domain (so https://nginx.lukeaddison.co.uk/oauth2/callback for me).
For other providers the configuration (especially the requirement to whitelist redirect URIs) should be similar. Whichever provider you use the setup process should return a Client ID and Client Secret which you should remember for a later step.
We can now enforce that access to the Nginx service be authenticated using our OIDC provider. Using the discovery URL (supported by most providers) we can retrieve the information required to configure Istio.
# https://developers.google.com/identity/protocols/oauth2/openid-connect#discovery
OIDC_DISCOVERY_URL="https://accounts.google.com/.well-known/openid-configuration"
OIDC_DISCOVERY_URL_RESPONSE=$(curl $OIDC_DISCOVERY_URL)
OIDC_ISSUER_URL=$(echo $OIDC_DISCOVERY_URL_RESPONSE | jq -r .issuer)
OIDC_JWKS_URI=$(echo $OIDC_DISCOVERY_URL_RESPONSE | jq -r .jwks_uri)
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- issuer: $OIDC_ISSUER_URL
jwksUri: $OIDC_JWKS_URI
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
action: CUSTOM
provider:
# Extension provider configured when we installed Istio
name: oauth2-proxy
rules:
- {}
EOFThe RequestAuthentication resource says that if a request to the ingress gateway contains a bearer token in the Authorization header then it must be a valid JWT signed by the specified OIDC provider. Istio will concatenate the iss and sub fields of the JWT with a / separator which will form the principal of the request. The AuthorizationPolicy says to contact oauth2-proxy for authorization decisions.
Since we have not deployed oauth2-proxy yet, visiting your domain again should now show: RBAC: access denied, so the final thing we need to do is to deploy oauth2-proxy to manage the OIDC flow, retrieve a JWT and inject it into each request.
oauth2-proxy
Envoy supports pluggable functionality known as filters which can be chained together and affect how requests are handled. One of Istio’s main roles is to configure these filters across a fleet of Envoys so that they form a mesh, supporting high-level APIs such as VirtualService and DestinationRule for operators to declare their desired mesh behaviour. The extension provider configuration we specified when we first installed Istio, together with the AuthorizationPolicy configuration above, configures the external authorization HTTP filter which is responsible for calling out to oauth2-proxy:
Before Istio 1.9, the same external authorization configuration could be supplied by applying an EnvoyFilter
# Set your Client ID and Client Secret from your OIDC provider setup
CLIENT_ID=""
CLIENT_SECRET=""
# https://oauth2-proxy.github.io/oauth2-proxy/docs/configuration/overview/#generating-a-cookie-secret
COOKIE_SECRET=$(openssl rand -base64 32 | tr -- '+/' '-_')
kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: oauth2-proxy
labels:
istio-injection: enabled
---
apiVersion: v1
kind: Secret
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
stringData:
OAUTH2_PROXY_CLIENT_ID: $CLIENT_ID
OAUTH2_PROXY_CLIENT_SECRET: $CLIENT_SECRET
OAUTH2_PROXY_COOKIE_SECRET: $COOKIE_SECRET
---
apiVersion: v1
kind: Service
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
spec:
selector:
app: oauth2-proxy
ports:
- name: http
port: 4180
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: oauth2-proxy
namespace: oauth2-proxy
spec:
selector:
matchLabels:
app: oauth2-proxy
template:
metadata:
labels:
app: oauth2-proxy
spec:
containers:
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.2.0
args:
- --provider=oidc
- --cookie-secure=true
- --cookie-samesite=lax
- --cookie-refresh=1h
- --cookie-expire=4h
- --cookie-name=_oauth2_proxy_istio_ingressgateway
- --set-authorization-header=true
- --email-domain=*
- --http-address=0.0.0.0:4180
- --upstream=static://200
- --skip-provider-button=true
- --whitelist-domain=$NGINX_DOMAIN
- --oidc-issuer-url=$OIDC_ISSUER_URL
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: OAUTH2_PROXY_CLIENT_ID
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: OAUTH2_PROXY_CLIENT_SECRET
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: oauth2-proxy
key: OAUTH2_PROXY_COOKIE_SECRET
resources:
requests:
cpu: 10m
memory: 100Mi
ports:
- containerPort: 4180
protocol: TCP
readinessProbe:
periodSeconds: 3
httpGet:
path: /ping
port: 4180
EOFNote that the above configuration tells oauth2-proxy to store session state as a browser cookie. This has the advantage of being stateless but can lead to large HTTP headers if the JWT returned from the OIDC flow is large; an alternative is to use Redis which adds a further operational burden but the size of the cookie is small and constant.
Visiting Nginx again you should be redirected to your OIDC provider. After signing in successfully oauth2-proxy should set an encrypted cookie which on subsequent requests will be decrypted to a JWT and attached as the Authorization header which Istio can validate.
If you want to authenticate multiple services you may want to configure the --cookie-domain and --whitelist-domain flags on the oauth2-proxy Deployment to include multiple subdomains. For example, I could set both of those flags to .lukeaddison.co.uk and then the cookie would be sent for any subdomain of lukeaddison.co.uk meaning I would only need to sign in once. To see the full set of configuration options go here.
Another powerful use case is to combine the OIDC authentication configuration with Istio’s ability to proxy to external services. For example, if I have a service running outside of Kubernetes but that does not have its own identity-aware authentication mechanism, Istio could be used as a reverse proxy to configure access to that service in a similar way to if it was running within the mesh.
Authorization
With the above configuration in place, we can now make further authorization decisions based on the attached JWT and corresponding claims. For example, to ensure that the user signed into our configured OAuth application (instead of another Google one) we can restrict access based on the audience claim. This is straightforward since the audience claim must contain our Client ID:
# Use of request.auth.audiences is currently not supported with CUSTOM action so we define the check
# as a separate AuthorizationPolicy
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: istio-ingressgateway-audience
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
rules:
- when:
- key: request.auth.audiences
values:
- $CLIENT_ID
EOFSince oauth2-proxy is making a decision about whether to allow a request depending on the existence of a JWT stored in an encrypted cookie, it shouldn’t be possible for a user to gain access using a JWT from a different source. Nevertheless, representing the expected value natively in Istio provides defence in depth.
We can also configure the ingress gateway Envoys to forward the Authorization header to our upstream service to allow for service specific policy (the following assumes that the email claim is present on the JWT):
EMAIL_ADDRESS="luke.addison@jetstack.io"
kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
selector:
matchLabels:
app: istio-ingressgateway
jwtRules:
- issuer: $OIDC_ISSUER_URL
jwksUri: $OIDC_JWKS_URI
# Forward JWT to Nginx sidecar
forwardOriginalToken: true
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: nginx
namespace: nginx
spec:
selector:
matchLabels:
app: nginx
jwtRules:
- issuer: $OIDC_ISSUER_URL
jwksUri: $OIDC_JWKS_URI
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: nginx
namespace: nginx
spec:
selector:
matchLabels:
app: nginx
rules:
- when:
- key: request.auth.audiences
values:
- $CLIENT_ID
- key: request.auth.claims[email]
values:
- $EMAIL_ADDRESS
EOFRestricting access to an email address which differs from yours should give RBAC: access denied as before. Unfortunately, currently only string and string list claims are extracted from the JWT, so in particular the boolean email_verified claim (signifying whether the provider took steps to ensure the email address was controlled by the end user) cannot be matched on. However, as described in the Claim Stability and Uniqueness section of the OIDC specification, the iss and sub claims used together are the only claims that can provide a stable identifier for a user, so if that is the goal then they should be used instead of the email claim. For more details on what is supported by AuthorizationPolicy see the documentation here and here.
The Dex example-app is a useful tool for retrieving a JWT locally to see the claims your identity provider returns for a particular set of scopes. Note the scopes requested by oauth2-proxy by default. By configuring oauth2-proxy to request different scopes, you can adjust the claims that are present on the returned JWT and thus the attributes that can be matched on for authorization decisions. Returning group membership for example allows access to particular services to be granted and revoked by simply moving users within your provider, without any changes to the Istio configuration.
As we have demonstrated, a really powerful aspect of this is that our backend service can be completely unaware that OIDC is being used and does not need to support it itself. However, if the service has support for parsing the JWT, then it can also be used to authorize granular access to different features of the service.
Cleanup
gcloud container clusters delete istio --asyncFuture
Another nascent project in this area is authservice which provides an alternative implementation of an external authorization endpoint, specifically for OIDC authentication.
Going with the theme of WASM extensibility in Istio, managing the OIDC flow may be a good candidate as a WASM extension so that Envoy no longer needs to call out to an external authorization implementation.
Get in Touch
TLS Protect for Kubernetes integrates directly with Istio for managing the security layer of your mesh at scale. If you found this post useful we’d encourage you to check it out, contact us directly to speak to an expert on service mesh security or download our guide to Zero Trust cloud-native infrastructure.