OIDC Multi-provider Support in Istio

This blog discusses implementing OIDC (OpenID Connect) multi-provider support in Istio for a Jetstack Consult customer. It outlines the challenges faced, including the need for multiple IDP (Identity Provider) providers, excluding specific paths like health checks, and managing OAuth2 proxy instances efficiently. The article provides a detailed walkthrough of the solution, including configuring OAuth2 proxy for each IDP, setting up Istio components, and handling common setup issues. It also discusses security considerations and the viability of the solution in a RedHat Service Mesh. The blog concludes by highlighting the extensibility of Istio's extAuthz mesh extension for authenticating multiple IDP providers.

Introduction

In this article we’ll walk through the challenges faced when building an OIDC session initiation solution with Istio for a customer. We started from the basis of another Jetstack blogpost walkthrough. In that article a test dex OIDC identity provider was deployed in-cluster. We’ve shown how it can be used with a real external idp and some other refinement that you might need in a production setting. It would help to read up on the previous work to get more familiar with what it’s presenting but I’ve included a small recap in this article in the recap section for the impatient. Firstly let’s explain the motivating use-case which brought us to write this post.

OIDC platform team requirements

One of our subscription clients at Jetstack Consult reached out for help in adapting the patterns presented by my esteemed colleague as a platform enhancement to reduce developer toil. This customer had specific needs that require a little adapting of the basic pattern presented in 1 These specific use-case points are:

Multiple IDP providers (eg. Ping and Microsoft AD) used by their different application teams.
Ability to exclude certain paths like health check
Running in Openshift
Want to deploy and manage the least amount of oauth2-proxy instances The developer teams were the clients of the platform team we were working with and each had their own bespoke authentication identity provider. The developers could either implement the login flow using libraries in their code or others leveraging distinct oauth2-proxy services for each namespace. One of the key benefits of a platform like Istio is in sharing common operational concerns to get more security and productivity benefits. In case you’re new to Istio and stumbled on this article, I recommend reading our introduction to Istio 4. While Istio comes with lots of great benefits out of the box. Sadly, the OIDC support isn’t really a first class support and requires quite a bit of custom work.

Kubernetes Subscriptions

Partner with experts to verify, maintain and optimize your Kubernetes production environment

Browse subscriptions

Recap of OIDC flow and previous work

In most setups of this kind, applications are configure to follow the Open ID Connect code flow. The oauth2-proxy will takes the role of an authenticating middleware in charge of exchanging an authorization code it obtains from the Identity provider. This consent is obtained by a login and/or consent screen and the middleware is authenticated to the iDP with a client secret to keep a long term session. In practice this means the first step we need is setup the Idp by creating an “App registration” that would need to whitelist the redirect URL which will be requested by the oauth2-proxy.

^{An example application integration in Azure AD}

When you create this application registration you will obtain a Client ID and can generate a client secret. The app registration will also provide you with the issuer and discovery metadata addresses. All these outputs will be used as input configuration to the oauth2-proxy to allow the authentication code flow to work properly.

The last piece of the puzzle is the plumbing by Istio to initiate the login flow by calling the oauth2-proxy and only allow successful login requests for the correct Idp, in the original solution 1 we were simply calling out to to the custom Authorizationpolicy extauthz provider in the Istio Ingress Gateway and as a defense-in-depth measure re-verifying every JWT at each micro-service using the RequestAuthentication Istio functionality. All extAuthz Istio/Envoy extensions are an envoy request filter that pauses incoming requests and does a separate HTTP call for authentication. The convention is that if, and only if, such service returns 200 it authorizes the request. The oauth2-proxy integration leverages the mechanism to perform the whole login flow (or check if already logged in) during that Pause and return 200 on successful login.

^{ExtAuthz Initiated at Gateway as shown in [1](}^{https://venafi.com/blog/istio-oidc/}⁾

Proposed solution walkthrough

The naive way to extend the above solution is to just create another extAuthz provider on the gateway for each idp, but this does not work, because it is not allowed to have multiple extAuthz providers on the same envoy proxy.

Processed authorization policy: failed to parse CUSTOM action, will generate a deny all config: only 1 provider can be used per workload, found multiple providers: [oauth2-proxy-ping oauth2-proxy-ad]

You would encounter the above error if you try to setup multiple extAuthz provider for a single workload or ingress gateway.

This guides our design to look like this:

^{ExtAuthz done at each microservice}

The below will talk through the implementation steps for two real demo applications that realise the above. Another possible alternative is to have a dedicated Istio api gateway per Extauthz provider, and perform the ExtAuthz at the gateway level. As this is a more resource heavy solution we will focus on the above design. Performing authorisation at the microservice level would also allow for internal service account authentication verification as well.

1. First we will prepare some domains and client applications in Ping and microsoft AD

export HTTPBIN_HOST=oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net
export BOOKINFO_HOST=oidc-azure-bookinfo.houssem-el-fekih-gcp.jetstacker.net
export PING_CLIENT_ID="REDACTED"
export PING_CLIENT_SECRET="REDACTED"
export PING_ISSUER_URL=https://auth.pingone.eu/REDACTED/as
export PING_JWKS_URL=https://auth.pingone.eu/REDACTED/as/jwks

export AZURE_CLIENT_ID="REDACTED"
export AZURE_CLIENT_SECRET='REDACTED'
etc..

2. Install cert-manager with helm

helm install cert-manager jetstack/cert-manager \                                                                                   
  --namespace cert-manager \
  --create-namespace \      
  --version v1.12.3 \
   --set installCRDs=true

3. Assuming Istio is already installed, setup a DNS entry with the appropriate host to your Istio ingress gateway, this is needed for the letsencrypt cert generation to have secure authentication.

4. Install the httpbin application, gateway and obtaining a cert with let’s encrypt

### Gateway httpbin + cert
kubectl label ns default istio-injection=enabled
kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml # httpbin app from the istio samples
envsubst <<"EOF" |  kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: istio-http-issuer
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: example-issuer-account-key
    solvers:
    - http01:
        ingress:
          class: istio
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ingress-cert
  namespace: istio-system
spec:
  secretName: ingress-cert
  dnsNames:
  - $HTTPBIN_HOST
  issuerRef:
    name: istio-http-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - "$HTTPBIN_HOST"
    tls:
      mode: SIMPLE
      credentialName: ingress-cert
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace:  istio-system
spec:
  hosts:
  - "*"
  gateways:
  - httpbin-gateway
  http:
  - route:
    - destination:
        host: httpbin.default.svc.cluster.local
        port:
          number: 8000
EOF

5. Running and configuring oauth2 proxy for ping using the following helm values

6. Use the below values for oauth2-proxy, you can use envsubst to pass in the idp specific bits

cat generic-values.yaml | envsubst > ping-values.yaml
## @section OAuth2 Proxy configuration parameters
## 

## Configuration section
##
configuration:
  ## @param configuration.clientID OAuth client ID
  ##
  clientID: "${PING_CLIENT_ID}"
  ## @param configuration.clientSecret OAuth client secret
  ##
  clientSecret: "${PING_CLIENT_SECRET}"
  ## Create a new secret with the following command openssl rand -base64 32 | head -c 32 | base64
  ## Use an existing secret for OAuth2 credentials (see secret.yaml for required fields)
  ##
  ## @param configuration.cookieSecret OAuth cookie secret
  ## don't reuse in real scenario
  cookieSecret: "SWpROHZaSjdxeFpSVEhIbVE2cHR1VlRyb3JzMmhld0E=" 

  content: |
    email_domains = [ "*" ]
    upstreams = [ "static://200" ]

  ## @param configuration.oidcIssuerUrl OpenID Connect issuer URL
  oidcIssuerUrl: "$PING_ISSUER_URL"

  ## @param configuration.whiteList Allowed domains for redirection after authentication. Prefix domain with a . or a *. to allow subdomains
  whiteList: "https://${HTTPBIN_HOST}"

extraArgs:
- --provider
- oidc
- --skip-provider-button

7. Install the oauth2-proxy:

helm upgrade -n oauth2-proxy -i oauth2-proxy-ping bitnami/oauth2-proxy -f ping-values.yaml –create-namespace

8. Repeat with the Bookinfo application and Azure AD (using the correct client secret etc..)

9. Add an “extauthz provider” for both oauth2 proxy(s)

istioctl install  --skip-confirmation -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: minimal
  hub: gcr.io/istio-testing
  tag: latest
  meshConfig:
    extensionProviders:
    - name: oauth2-proxy-azure
      envoyExtAuthzHttp:
        service: oauth2-proxy-azure.oauth2-proxy.svc.cluster.local
        port: 80
        includeRequestHeadersInCheck:
        - cookie
        headersToUpstreamOnAllow:
        - authorization
        headersToDownstreamOnDeny:
        - set-cookie
    - name: oauth2-proxy-ping
      envoyExtAuthzHttp:
        service: oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
        port: 80
        includeRequestHeadersInCheck:
        - cookie
        headersToUpstreamOnAllow:
        - authorization
        headersToDownstreamOnDeny:
        - set-cookie
    accessLogFile: /dev/stdout
    enableTracing: true
  components:
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
EOF

I’ve seen much more complex versions of this extAuthZ configuration in various guides but this is enough to get most setups working properly without adding unnecessary complexity!

10. Istio Authn and Authz configuration

envsubst <<"EOF" |  kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: app-req-auth-ping
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - issuer: ${PING_ISSUER_URL}
    jwksUri: ${PING_JWKS_URL}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: app-auth-policy-ping
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  action: CUSTOM
  provider:
    # Extension provider configured when we installed Istio
    name: oauth2-proxy-ping
  rules:
  - {}

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: app-auth-policy-ad
  namespace: default
spec:
  selector:
    matchLabels:
      app: productpage
  action: CUSTOM
  provider:
    # Extension provider configured when we installed Istio
    name: oauth2-proxy-azure
  rules: 
  - {}
---

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: app-auth-policy-ad
  namespace: default
spec:
  selector:
    matchLabels:
      app: productpage
  jwtRules:
  - issuer: ${AZURE_ISSUER_URL}
    jwksUri: ${AZURE_JWKS_URL}
EOF

We can now login to either Azure or Ping by accessing the appropriate paths: Login through the ping directory is initiating when accessing oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/

^{Ping login demonstration through Istio ExtAuthz :)}

Login through Azure is intiated when accessing https://oidc-azure-bookinfo.houssem-el-fekih-gcp.jetstacker.net/ and this has been tested to work properly

Common setup failures

Bad redirect URL:

The redirect url is inferred automatically by oauth2-proxy to be https://application-domain/oauth2/callback but the backing identity provider (AD or Ping) could be operating another whitelist of the domains allowed for that application, make sure these match

Error creating session during OAuth2 callback: neither the id_token nor the profileURL set an email in oauth2-proxy:

Add the email scope to your identity provider’s application integration

How do I sign out to retest:

Aside from using incognito mode, you can access the /oauth2/sign_out endpoint to force oauth2-proxy to drop that session, however you will also need to remove any cache on the identity provider to fully invalidate the session or you will be immediately logged back in more details in 5

Cache settings:

Ensure that your oauth2-proxy cache settings match the session time on the identity provider side with the --cookie-expire

Hardening:

Our configuration is minimal and is about the mechanics of it, some of the hardening you should look into:

S256 challenge method with --code-challenge-method which requires an extra secret
cookie domain filter with --cookie-domain
secure cookie
force https
Because you have request authentication set as well, you can add claim checks like audience for more security

These are just a few of the most critical ones, the oauth2-proxy has many different security mechanisms that you should explore.

Discussion of tradeoffs

As we demonstrated above we’ve accomplished our primary objective, but there are a few questions that naturally arise in terms of the Istio integration. oauth2-proxy best practices such as scalability, cache/persistence (which are typically handled by a redis service) are not the focus of our article but some security tips are given in the previous section.

If OIDC is managed by mesh providers, per microservice, how do we control the behavior for various paths such as for example allowing health checks through?
The communication with oauth2-proxy is obviously sensitive and could be the target for an attacker if somehow communication is intercepted, could we ensure TLS communication at that leg? Would running oauth2-proxy through the mesh work?

Health check exclusion

1. Excluding /status/200 to simulate a health check on httpbin works but we have to be careful how we configure it, if we try to create a separate AuthorizationPolicy like so:

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin-allow-healthz
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        requestPrincipals: ["*"]
  - to:
    - operation:
        paths: ["/status/200"]

Then it will not take effect because the CUSTOM action takes precedence and denies it, for reference check this chart from the Istio docs:

^{CUSTOM policies take precedence}

Instead, we should modify the CUSTOM authorization policy and directly specify which paths to perform the login for as shown below, this requires us to put the list of paths excluded from forwarding to the OIDC provider , hence the notPaths clause.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: app-auth-policy-ping
  namespace: default
spec:
  selector:
    matchLabels:
      app: httpbin
  action: CUSTOM
  provider:
    # Extension provider configured when we installed Istio
    name: oauth2-proxy-ping
  rules:
  - to:
    - operation:
        notPaths: ["/status/200"]

We can now simply curl that endpoint without a JWT and we would be allowed through:

curl -I https://oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/status/200
    HTTP/2 200 
    server: istio-envoy
    date: Mon, 09 Jan 2023 13:06:15 GMT
    content-type: text/html; charset=utf-8
    access-control-allow-origin: *
    access-control-allow-credentials: true
    content-length: 0
    x-envoy-upstream-service-time: 33

Any other paths still causes a redirect response to the ping login screen:

curl -I https://oidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net/headers
    HTTP/2 302 
    location: https://auth.pingone.eu/36e420ff-0151-44ea-9939-ed49e25c9b6e/as/authorize?approval_prompt=force&client_id=33db4b35-25bf-42b5-affd-3401c91ab4c6&redirect_uri=https%3A%2F%2Foidc-ping-httpbin.houssem-el-fekih-gcp.jetstacker.net%2Foauth2%2Fcallback&response_type=code&scope=openid+email+profile&state=h3wWQy_P_xAWlK0g9PRtJqnRFDy9FZDtpUcyDUhJRAY%3A%2Fheaders
    set-cookie: _oauth2_proxy_csrf=7yS5QsjYyoRoMMTN8T1DwuwZ5vdSNaiEjtpxDLcaOOMTNtrvL_9NVXFnNv8PYDF4qgxK6sMUxlizXsjtFcEOakx5ASSbc068ienX_E73snjNC3nTVw4uwUk=|1673269604|TQWpzmvyQYwmpdCR_vsZ0jbn8iiSWZTR5UTXBH7PXbY=; Path=/; Expires=Mon, 09 Jan 2023 13:21:44 GMT; HttpOnly; Secure
    date: Mon, 09 Jan 2023 13:06:44 GMT
    server: istio-envoy
    x-envoy-upstream-service-time: 3

mTLS to `oauth2-proxy`

The oauth2 proxy can join the mesh, if we label the oauth2-proxy namespace and restart kubectl labels ns oauth2-proxy istio-injection=enabled && kubectl rollout restart deploy -n oauth2-proxy The oauth2-proxies have joined the mesh successfully and the authentication is working fine through mtls We can see that the extAuthz block in the listener config is talking to this particular cluster:

- name: envoy.filters.http.ext_authz
  typedConfig:
    '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
    filterEnabledMetadata:
      filter: envoy.filters.http.rbac
      path:
      - key: istio_ext_authz_shadow_effective_policy_id
      value:
        stringMatch:
          prefix: istio-ext-authz
    httpService:
      authorizationRequest:
        allowedHeaders:
          patterns:
          - exact: cookie
            ignoreCase: true
      authorizationResponse:
        allowedClientHeaders:
          patterns:
          - exact: set-cookie
            ignoreCase: true
        allowedUpstreamHeaders:
          patterns:
          - exact: authorization
            ignoreCase: true
      serverUri:
        cluster: outbound|80||oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
        timeout: 600s
        uri: http://oauth2-proxy-ping.oauth2-proxy.svc.cluster.local
    transportApiVersion: V3

This works as expected where the oauth2-proxy endpoints have a cluster setup and if it joins the mesh, the cluster will automatically be configured to use the ISTIO_MUTUAL method of authentication.

Viability of solution in RedHat service mesh

A note for users of the Redhat service mesh product is that being able to use these CUSTOM auth policies requires modifying the “meshconfig” for the mesh. The SMCP object did not allow us to set it out of the box, on the version we tested. Our customer has raised a support request with RedHat and the ability to add external auth providers by setting mesh config has been added to techPreview on SMCP 2.3 and this is now GA in version 2.4 3 We did also successfully test manually injecting the extAuthZ blocks as envoyFilter blocks targeting each micro-service. As you may know the envoyFilter is a break glass API which will force you to keep maintaining this configuration across each Istio version that you upgrade to.

Conclusion

The Istio project has an incredible array of extension points, and we have shown that the extAuthz mesh extension can be used for authenticating multiple IDP providers according to the needs of your developer teams and their various micro-services. We have clarified a few of the undocumented corner cases of using this extension and this approach can be templated for various services for a platform managed OIDC mechanism. We have not touched on further oauth2-proxy options or studied its performance characteristics on this blog post, but given that it’s an established, long running project, it should be possible to scale and adapt it to your specific workflow. It is unfortunate to see that this extAuthZ api has not graduated to a real Istio API yet. However we’ve tested that the meshConfig support has been added to Openshift making this solution viable for this platform albeit complex to configure. Venafi Jetstack Consult is working with many clients who are looking to take advantage of deploying Istio Service Mesh in their organisation. If you’re interested in learning more about our expertise and how we can help to modernise your Kubernetes investment, reach out to us here or check out more of our blogs. Furthermore, we have produced a really insightful technical paper on how to plan and deploy a fully operational Istio service mesh using open source best practices alongside cloud native solutions for advanced security. You can download this paper here.

Cloud Native Consulting Services

When our experts are your experts, you can make the most of Kubernetes

Find an expert

References

[1] https://venafi.com/blog/istio-oidc/
[2] https://developer.okta.com/blog/2017/07/25/oidc-primer-part-1
[3] https://github.com/maistra/istio-operator/pull/1136
[4] https://venafi.com/blog/service-meshes-a-deeper-introduction/
[5] https://oauth2-proxy.github.io/oauth2-proxy/docs/features/endpoints#sign-out

OIDC Multi-provider Support in Istio

Introduction

OIDC platform team requirements

Partner with experts to verify, maintain and optimize your Kubernetes production environment

Recap of OIDC flow and previous work

Proposed solution walkthrough

Common setup failures

Bad redirect URL:

Error creating session during OAuth2 callback: neither the id_token nor the profileURL set an email in oauth2-proxy:

How do I sign out to retest:

Cache settings:

Hardening:

Discussion of tradeoffs

Health check exclusion

mTLS to `oauth2-proxy`

Viability of solution in RedHat service mesh

Conclusion

When our experts are your experts, you can make the most of Kubernetes

References

Products

Solution Offerings

Solutions by Industry

Consulting

Resources

Ecosystem

Open Source

Company

Introduction

OIDC platform team requirements

Partner with experts to verify, maintain and optimize your Kubernetes production environment

Recap of OIDC flow and previous work

Proposed solution walkthrough

Common setup failures

Bad redirect URL:

Error creating session during OAuth2 callback: neither the id_token nor the profileURL set an email in oauth2-proxy:

How do I sign out to retest:

Cache settings:

Hardening:

Discussion of tradeoffs

Health check exclusion

mTLS to oauth2-proxy

Viability of solution in RedHat service mesh

Conclusion

When our experts are your experts, you can make the most of Kubernetes

References

Products

Solution Offerings

Solutions by Industry

Consulting

Resources

Ecosystem

Open Source

Company

mTLS to `oauth2-proxy`