This blog discusses implementing OIDC (OpenID Connect) multi-provider support in Istio for a Jetstack Consult customer. It outlines the challenges faced, including the need for multiple IDP (Identity Provider) providers, excluding specific paths like health checks, and managing OAuth2 proxy instances efficiently. The article provides a detailed walkthrough of the solution, including configuring OAuth2 proxy for each IDP, setting up Istio components, and handling common setup issues. It also discusses security considerations and the viability of the solution in a RedHat Service Mesh. The blog concludes by highlighting the extensibility of Istio's extAuthz mesh extension for authenticating multiple IDP providers.
Introduction
In this article we’ll walk through the challenges faced when building an OIDC session initiation solution with Istio for a customer. We started from the basis of another Jetstack blogpost walkthrough. In that article a test dex OIDC identity provider was deployed in-cluster. We’ve shown how it can be used with a real external idp and some other refinement that you might need in a production setting. It would help to read up on the previous work to get more familiar with what it’s presenting but I’ve included a small recap in this article in the recap section for the impatient. Firstly let’s explain the motivating use-case which brought us to write this post.
OIDC platform team requirements
One of our subscription clients at Jetstack Consult reached out for help in adapting the patterns presented by my esteemed colleague as a platform enhancement to reduce developer toil. This customer had specific needs that require a little adapting of the basic pattern presented in 1 These specific use-case points are:
- Multiple IDP providers (eg. Ping and Microsoft AD) used by their different application teams.
- Ability to exclude certain paths like health check
- Running in Openshift
- Want to deploy and manage the least amount of oauth2-proxy instances The developer teams were the clients of the platform team we were working with and each had their own bespoke authentication identity provider. The developers could either implement the login flow using libraries in their code or others leveraging distinct oauth2-proxy services for each namespace. One of the key benefits of a platform like Istio is in sharing common operational concerns to get more security and productivity benefits. In case you’re new to Istio and stumbled on this article, I recommend reading our introduction to Istio 4. While Istio comes with lots of great benefits out of the box. Sadly, the OIDC support isn’t really a first class support and requires quite a bit of custom work.
Partner with experts to verify, maintain and optimize your Kubernetes production environment
Recap of OIDC flow and previous work
In most setups of this kind, applications are configure to follow the Open ID Connect code flow. The oauth2-proxy will takes the role of an authenticating middleware in charge of exchanging an authorization code it obtains from the Identity provider. This consent is obtained by a login and/or consent screen and the middleware is authenticated to the iDP with a client secret to keep a long term session. In practice this means the first step we need is setup the Idp by creating an “App registration” that would need to whitelist the redirect URL which will be requested by the oauth2-proxy.
![](https://cdn.venafi.com/994513b8-133f-0003-9fb3-9cbe4b61ffeb/715d6ab9-1500-4ebc-82ea-de11e8297db1/image4.png?fm=webp&q=85)
An example application integration in Azure AD
When you create this application registration you will obtain a Client ID and can generate a client secret. The app registration will also provide you with the issuer and discovery metadata addresses. All these outputs will be used as input configuration to the oauth2-proxy to allow the authentication code flow to work properly.
The last piece of the puzzle is the plumbing by Istio to initiate the login flow by calling the oauth2-proxy and only allow successful login requests for the correct Idp, in the original solution 1 we were simply calling out to to the custom Authorizationpolicy extauthz provider in the Istio Ingress Gateway and as a defense-in-depth measure re-verifying every JWT at each micro-service using the RequestAuthentication
Istio functionality. All extAuthz Istio/Envoy extensions are an envoy request filter that pauses incoming requests and does a separate HTTP call for authentication. The convention is that if, and only if, such service returns 200 it authorizes the request. The oauth2-proxy integration leverages the mechanism to perform the whole login flow (or check if already logged in) during that Pause and return 200 on successful login.
![](https://cdn.venafi.com/994513b8-133f-0003-9fb3-9cbe4b61ffeb/6466b76f-b418-4941-ba57-815249029624/image1.png?fm=webp&q=85)
ExtAuthz Initiated at Gateway as shown in [1](https://venafi.com/blog/istio-oidc/)
Proposed solution walkthrough
The naive way to extend the above solution is to just create another extAuthz provider on the gateway for each idp, but this does not work, because it is not allowed to have multiple extAuthz providers on the same envoy proxy.
Processed authorization policy: failed to parse CUSTOM action, will generate a deny all config: only 1 provider can be used per workload, found multiple providers: [oauth2-proxy-ping oauth2-proxy-ad]
You would encounter the above error if you try to setup multiple extAuthz provider for a single workload or ingress gateway.
This guides our design to look like this:
![](https://cdn.venafi.com/994513b8-133f-0003-9fb3-9cbe4b61ffeb/1d41796b-2b71-4a95-9a41-7cb886e7971d/image5.png?fm=webp&q=85)
ExtAuthz done at each microservice
The below will talk through the implementation steps for two real demo applications that realise the above. Another possible alternative is to have a dedicated Istio api gateway per Extauthz provider, and perform the ExtAuthz at the gateway level. As this is a more resource heavy solution we will focus on the above design. Performing authorisation at the microservice level would also allow for internal service account authentication verification as well.
1. First we will prepare some domains and client applications in Ping and microsoft AD
export HTTPBIN_HOST=oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net
export BOOKINFO_HOST=oidc-azure-bookinfo.houssem-el-fekih-gcp.jetstacker.net
export PING_CLIENT_ID="REDACTED"
export PING_CLIENT_SECRET="REDACTED"
export PING_ISSUER_URL=https://auth.pingone.eu/REDACTED/as
export PING_JWKS_URL=https://auth.pingone.eu/REDACTED/as/jwks
export AZURE_CLIENT_ID="REDACTED"
export AZURE_CLIENT_SECRET='REDACTED'
etc..
2. Install cert-manager with helm
helm install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.12.3 \
--set installCRDs=true
3. Assuming Istio is already installed, setup a DNS entry with the appropriate host to your Istio ingress gateway, this is needed for the letsencrypt cert generation to have secure authentication.
4. Install the httpbin application, gateway and obtaining a cert with let’s encrypt
### Gateway httpbin + cert
kubectl label ns default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml # httpbin app from the istio samples
envsubst <<"EOF" | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: istio-http-issuer
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: example-issuer-account-key
solvers:
- http01:
ingress:
class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: ingress-cert
namespace: istio-system
spec:
secretName: ingress-cert
dnsNames:
- $HTTPBIN_HOST
issuerRef:
name: istio-http-issuer
kind: ClusterIssuer
group: cert-manager.io
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: httpbin-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- port:
number: 443
name: https
protocol: HTTPS
hosts:
- "$HTTPBIN_HOST"
tls:
mode: SIMPLE
credentialName: ingress-cert
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: httpbin
namespace: istio-system
spec:
hosts:
- "*"
gateways:
- httpbin-gateway
http:
- route:
- destination:
host: httpbin.default.svc.cluster.local
port:
number: 8000
EOF
5. Running and configuring oauth2 proxy for ping using the following helm values
6. Use the below values for oauth2-proxy, you can use envsubst to pass in the idp specific bits
cat generic-values.yaml | envsubst > ping-values.yaml
## @section OAuth2 Proxy configuration parameters
##
## Configuration section
##
configuration:
## @param configuration.clientID OAuth client ID
##
clientID: "${PING_CLIENT_ID}"
## @param configuration.clientSecret OAuth client secret
##
clientSecret: "${PING_CLIENT_SECRET}"
## Create a new secret with the following command openssl rand -base64 32 | head -c 32 | base64
## Use an existing secret for OAuth2 credentials (see secret.yaml for required fields)
##
## @param configuration.cookieSecret OAuth cookie secret
## don't reuse in real scenario
cookieSecret: "SWpROHZaSjdxeFpSVEhIbVE2cHR1VlRyb3JzMmhld0E="
content: |
email_domains = [ "*" ]
upstreams = [ "static://200" ]
## @param configuration.oidcIssuerUrl OpenID Connect issuer URL
oidcIssuerUrl: "$PING_ISSUER_URL"
## @param configuration.whiteList Allowed domains for redirection after authentication. Prefix domain with a . or a *. to allow subdomains
whiteList: "https://${HTTPBIN_HOST}"
extraArgs:
- --provider
- oidc
- --skip-provider-button
7. Install the oauth2-proxy:
helm upgrade -n oauth2-proxy -i oauth2-proxy-ping bitnami/oauth2-proxy -f ping-values.yaml –create-namespace
8. Repeat with the Bookinfo application and Azure AD (using the correct client secret etc..)
9. Add an “extauthz provider” for both oauth2 proxy(s)
istioctl install --skip-confirmation -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
profile: minimal
hub: gcr.io/istio-testing
tag: latest
meshConfig:
extensionProviders:
- name: oauth2-proxy-azure
envoyExtAuthzHttp:
service: oauth2-proxy-azure.oauth2-proxy.svc.cluster.local
port: 80
includeRequestHeadersInCheck:
- cookie
headersToUpstreamOnAllow:
- authorization
headersToDownstreamOnDeny:
- set-cookie
- name: oauth2-proxy-ping
envoyExtAuthzHttp:
service: oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
port: 80
includeRequestHeadersInCheck:
- cookie
headersToUpstreamOnAllow:
- authorization
headersToDownstreamOnDeny:
- set-cookie
accessLogFile: /dev/stdout
enableTracing: true
components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
EOF
I’ve seen much more complex versions of this extAuthZ configuration in various guides but this is enough to get most setups working properly without adding unnecessary complexity!
10. Istio Authn and Authz configuration
envsubst <<"EOF" | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: app-req-auth-ping
namespace: default
spec:
selector:
matchLabels:
app: httpbin
jwtRules:
- issuer: ${PING_ISSUER_URL}
jwksUri: ${PING_JWKS_URL}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: app-auth-policy-ping
namespace: default
spec:
selector:
matchLabels:
app: httpbin
action: CUSTOM
provider:
# Extension provider configured when we installed Istio
name: oauth2-proxy-ping
rules:
- {}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: app-auth-policy-ad
namespace: default
spec:
selector:
matchLabels:
app: productpage
action: CUSTOM
provider:
# Extension provider configured when we installed Istio
name: oauth2-proxy-azure
rules:
- {}
---
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
name: app-auth-policy-ad
namespace: default
spec:
selector:
matchLabels:
app: productpage
jwtRules:
- issuer: ${AZURE_ISSUER_URL}
jwksUri: ${AZURE_JWKS_URL}
EOF
We can now login to either Azure or Ping by accessing the appropriate paths: Login through the ping directory is initiating when accessing oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/
![](https://cdn.venafi.com/994513b8-133f-0003-9fb3-9cbe4b61ffeb/c8e206b6-f861-4fea-b219-dd2a7615af6d/image2.gif?fm=webp&q=85)
Ping login demonstration through Istio ExtAuthz :)
Login through Azure is intiated when accessing https://oidc-azure-bookinfo.houssem-el-fekih-gcp.jetstacker.net/ and this has been tested to work properly
Common setup failures
Bad redirect URL:
The redirect url is inferred automatically by oauth2-proxy to be https://application-domain/oauth2/callback but the backing identity provider (AD or Ping) could be operating another whitelist of the domains allowed for that application, make sure these match
Error creating session during OAuth2 callback: neither the id_token nor the profileURL set an email in oauth2-proxy:
Add the email scope to your identity provider’s application integration
How do I sign out to retest:
Aside from using incognito mode, you can access the /oauth2/sign_out endpoint to force oauth2-proxy to drop that session, however you will also need to remove any cache on the identity provider to fully invalidate the session or you will be immediately logged back in more details in 5
Cache settings:
Ensure that your oauth2-proxy cache settings match the session time on the identity provider side with the --cookie-expire
Hardening:
Our configuration is minimal and is about the mechanics of it, some of the hardening you should look into:
- S256 challenge method with --code-challenge-method which requires an extra secret
- cookie domain filter with --cookie-domain
- secure cookie
- force https
- Because you have request authentication set as well, you can add claim checks like audience for more security
These are just a few of the most critical ones, the oauth2-proxy has many different security mechanisms that you should explore.
Discussion of tradeoffs
As we demonstrated above we’ve accomplished our primary objective, but there are a few questions that naturally arise in terms of the Istio integration. oauth2-proxy best practices such as scalability, cache/persistence (which are typically handled by a redis service) are not the focus of our article but some security tips are given in the previous section.
- If OIDC is managed by mesh providers, per microservice, how do we control the behavior for various paths such as for example allowing health checks through?
- The communication with oauth2-proxy is obviously sensitive and could be the target for an attacker if somehow communication is intercepted, could we ensure TLS communication at that leg? Would running oauth2-proxy through the mesh work?
Health check exclusion
1. Excluding /status/200 to simulate a health check on httpbin works but we have to be careful how we configure it, if we try to create a separate AuthorizationPolicy like so:
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: httpbin-allow-healthz
namespace: default
spec:
selector:
matchLabels:
app: httpbin
rules:
- from:
- source:
requestPrincipals: ["*"]
- to:
- operation:
paths: ["/status/200"]
Then it will not take effect because the CUSTOM action takes precedence and denies it, for reference check this chart from the Istio docs:
![](https://cdn.venafi.com/994513b8-133f-0003-9fb3-9cbe4b61ffeb/ca619ccf-47a8-42ca-948f-ffd7d9fc0174/image3.png?fm=webp&q=85)
CUSTOM policies take precedence
Instead, we should modify the CUSTOM authorization policy and directly specify which paths to perform the login for as shown below, this requires us to put the list of paths excluded from forwarding to the OIDC provider , hence the notPaths clause.
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: app-auth-policy-ping
namespace: default
spec:
selector:
matchLabels:
app: httpbin
action: CUSTOM
provider:
# Extension provider configured when we installed Istio
name: oauth2-proxy-ping
rules:
- to:
- operation:
notPaths: ["/status/200"]
We can now simply curl that endpoint without a JWT and we would be allowed through:
curl -I https://oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/status/200
HTTP/2 200
server: istio-envoy
date: Mon, 09 Jan 2023 13:06:15 GMT
content-type: text/html; charset=utf-8
access-control-allow-origin: *
access-control-allow-credentials: true
content-length: 0
x-envoy-upstream-service-time: 33
Any other paths still causes a redirect response to the ping login screen:
curl -I https://oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/headers
HTTP/2 302
location: https://auth.pingone.eu/36e420ff-0151-44ea-9939-ed49e25c9b6e/as/authorize?approval_prompt=force&client_id=33db4b35-25bf-42b5-affd-3401c91ab4c6&redirect_uri=https%3A%2F%2Foidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=h3wWQy_P_xAWlK0g9PRtJqnRFDy9FZDtpUcyDUhJRAY%3A%2Fheaders
set-cookie: _oauth2_proxy_csrf=7yS5QsjYyoRoMMTN8T1DwuwZ5vdSNaiEjtpxDLcaOOMTNtrvL_9NVXFnNv8PYDF4qgxK6sMUxlizXsjtFcEOakx5ASSbc068ienX_E73snjNC3nTVw4uwUk=|1673269604|TQWpzmvyQYwmpdCR_vsZ0jbn8iiSWZTR5UTXBH7PXbY=; Path=/; Expires=Mon, 09 Jan 2023 13:21:44 GMT; HttpOnly; Secure
date: Mon, 09 Jan 2023 13:06:44 GMT
server: istio-envoy
x-envoy-upstream-service-time: 3
mTLS to oauth2-proxy
The oauth2 proxy can join the mesh, if we label the oauth2-proxy namespace and restart kubectl labels ns oauth2-proxy istio-injection=enabled && kubectl rollout restart deploy -n oauth2-proxy
The oauth2-proxies have joined the mesh successfully and the authentication is working fine through mtls We can see that the extAuthz block in the listener config is talking to this particular cluster:
- name: envoy.filters.http.ext_authz
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
filterEnabledMetadata:
filter: envoy.filters.http.rbac
path:
- key: istio_ext_authz_shadow_effective_policy_id
value:
stringMatch:
prefix: istio-ext-authz
httpService:
authorizationRequest:
allowedHeaders:
patterns:
- exact: cookie
ignoreCase: true
authorizationResponse:
allowedClientHeaders:
patterns:
- exact: set-cookie
ignoreCase: true
allowedUpstreamHeaders:
patterns:
- exact: authorization
ignoreCase: true
serverUri:
cluster: outbound|80||oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
timeout: 600s
uri: http://oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
transportApiVersion: V3
This works as expected where the oauth2-proxy endpoints have a cluster setup and if it joins the mesh, the cluster will automatically be configured to use the ISTIO_MUTUAL
method of authentication.
Viability of solution in RedHat service mesh
A note for users of the Redhat service mesh product is that being able to use these CUSTOM
auth policies requires modifying the “meshconfig” for the mesh. The SMCP object did not allow us to set it out of the box, on the version we tested. Our customer has raised a support request with RedHat and the ability to add external auth providers by setting mesh config has been added to techPreview on SMCP 2.3 and this is now GA in version 2.4 3 We did also successfully test manually injecting the extAuthZ blocks as envoyFilter blocks targeting each micro-service. As you may know the envoyFilter is a break glass API which will force you to keep maintaining this configuration across each Istio version that you upgrade to.
Conclusion
The Istio project has an incredible array of extension points, and we have shown that the extAuthz mesh extension can be used for authenticating multiple IDP providers according to the needs of your developer teams and their various micro-services. We have clarified a few of the undocumented corner cases of using this extension and this approach can be templated for various services for a platform managed OIDC mechanism. We have not touched on further oauth2-proxy options or studied its performance characteristics on this blog post, but given that it’s an established, long running project, it should be possible to scale and adapt it to your specific workflow. It is unfortunate to see that this extAuthZ api has not graduated to a real Istio API yet. However we’ve tested that the meshConfig support has been added to Openshift making this solution viable for this platform albeit complex to configure. Venafi Jetstack Consult is working with many clients who are looking to take advantage of deploying Istio Service Mesh in their organisation. If you’re interested in learning more about our expertise and how we can help to modernise your Kubernetes investment, reach out to us here or check out more of our blogs. Furthermore, we have produced a really insightful technical paper on how to plan and deploy a fully operational Istio service mesh using open source best practices alongside cloud native solutions for advanced security. You can download this paper here.
When our experts are your experts, you can make the most of Kubernetes
References
[1] https://venafi.com/blog/istio-oidc/
[2] https://developer.okta.com/blog/2017/07/25/oidc-primer-part-1
[3] https://github.com/maistra/istio-operator/pull/1136
[4] https://venafi.com/blog/service-meshes-a-deeper-introduction/
[5] https://oauth2-proxy.github.io/oauth2-proxy/docs/features/endpoints#sign-out
![Summit 2024 Teaser Photo](/_next/image/?url=https%3A%2F%2Fcdn.venafi.com%2F994513b8-133f-0003-9fb3-9cbe4b61ffeb%2F537b500f-ee2a-44ba-82b4-d7be0b23506c%2Fsummit-promo-graphic-2.jpg&w=3840&q=75&dpl=dpl_9nMnPWFgUR3MNGqVKD9tfneGCtKt)
Machine Identity Security Summit 2024
Help us forge a new era of cybersecurity
☕ We're spilling all the machine identiTEA Oct. 1-3, but these insights are too valuable to just toss in the harbor! Browse the agenda and register now.