![DevOps and the Proliferation of Secrets [Keeping Data Safe] - cover graphic](/_next/image/?url=https%3A%2F%2Fcdn.venafi.com%2F994513b8-133f-0003-9fb3-9cbe4b61ffeb%2Fe0d304c4-8de6-4656-ba19-cbec96108b54%2Faid_field_img__8ffede2af7e0d3c609abad6adb1966cc.png%3Fw%3D864%26h%3D370%26fm%3Dwebp%26q%3D75%26fit%3Dcrop&w=3840&q=75&dpl=dpl_H1HoLru8zbnj4wwarzUVadLRmxGZ){kind=small}
‘Secrets’, in the context of cybersecurity and DevSecOps, describes the digital authentication credentials used in applications and services; anything you don’t want to share. The secrets you need to keep safe may include:
- Usernames and passwords (also auto-generated passwords)
- API tokens and other application keys/credentials (including within containers)
- SSH Keys
- Database and other system-to-system passwords
- Private certificates for secure communication, transmitting and receiving of data (TLS, SSL, etc.)
- Private encryption keys for systems like Pretty Good Privacy (PGP)
- RSA and other one-time password devices
- Keypad pin sequences
- Symmetric encryption keys
These secrets need to stay secret because they are the doors to your data—if the wrong person gets hold of them, you have the potential to have an unwelcome intruder in your systems wreaking whatever havoc they can. Applications leak secrets under all sorts of circumstances: applications frequently log configurations, leaving them in log files or centralized logging systems. Often secrets will be captured in exception tracebacks or crash reports sent to external monitoring systems, or they will be leaked via debugging endpoints and diagnostic pages after hitting an error. Here are some examples of what can happen when secrets aren’t properly kept:
- Uber hacked after leaving secrets on Github
- Viacom left secrets publicly accessible on Puppet
- Facebook stored passwords in plain text
- Accenture left secrets exposed on Amazon S3
- Pinterest built their own secrets management tool (Knox) in response to breaches
So, are you sure your secrets are safe? Can you answer these questions?
- Where are all my secrets?
- Who is accessing them and when?
- Are they still there?
- When did they get there?
- Have they changed?
- When were they removed?
- How do I rotate all of these secrets to match security policies?
You might now be thinking that it used to be easier to answer those questions. Historically the IP address was the unit of identity for security, and independent from any PKI identity. This was a workable approach when IPs were relatively static, and the rate of change was low. The challenge we are facing in the DevOps world is that our secrets are proliferating as a result of our desire to deploy changes more frequently. Our use of cloud and other modern technology practices (e.g. microservices, containers) makes our infrastructure more dynamic and ephemeral. We can’t use the IP address now because it’s no longer a static value from which we can base all our policies. It’s really hard for the technology teams to keep up even with the basic tasks of secrets generation, rotation, revocation, assignment, and sharing, particularly with all the other pressures on us to digitally transform, move from project to product, automate and evolve our cultures.
Causes of proliferation
- Machine Identity: The number of machines in organisations naturally proliferate through virtualization and containerization so these machines need their identities to be managed via secrets.
- Cloud and Serverless: Teams developing cloud-native applications need to use secrets to mediate potentially thousands of services—which also vary by platform, so the challenge becomes ever more complex in a hybrid or multi-cloud environment.
- Microservices: The modular services in a microservices environment are connected using HTTP or REST interfaces. This distributed model results in proliferation of interfaces and the communication between them generates several secrets management challenges. Additionally, each microservices module has its own database and credentials, thereby increasing the number of secrets to be managed.
- DevOps: As infrastructure squads, IT operations teams and developers build Continuous Integration and Continuous Delivery (CI/CD) platforms, they must also secure these deployment pipelines.
- AI/ML, IoT, Robotics and embedded devices: Burgeoning use of these new technologies compound the problem as every endpoint needs encryption and certificates too.
- Blockchain: This too creates an increase in the number of private keys compared to a typical application driving the need for enterprise wallets.
Zero Trust with cert-manager, Istio and Kubernetes
Bad behaviour: secrets sprawl
Secrets sprawl describes the surface area on which an organization’s secrets are stored along with the sheer volume of secrets as well as issues with secrets being duplicated and residing in multiple places—they have sprawled from their original and safe home. It is by no means uncommon for secrets to be stored in plain text in a variety of different places such as:
- Files on individuals’ devices
- Version control systems like GitHub*, Bitbucket or Azure DevOps
- Configuration management platforms like Chef, Puppet, Ansible or Terraform
- In CI/CD pipelines like Jenkins, TeamCity or Bamboo
- On a sticky note (!)
*GitHub has a Secret Audit tool that you may find useful if this is your version control tool of choice. You could also take a look at TruffleHog.
Best Practices for Secrets Management in DevSecOps
Secrets should:
- Never be stored in plain text
- Never be transmitted over a network unencrypted
- Have their value minimized
- Be frequently and regularly rotated
- Be as short-lived as possible
- Be automatically revoked where possible
- Have all lifecycle and access events recorded - fully auditable
- Distributed by an authoritative delegator
- Have limited operator access
- Be versioned more easily than revealing plain text
- Be mutually authenticated with related infrastructure components
- Use role-based access control where possible
The best current way to ensure you and your teams are following these best practices is to use a Secrets Management tool for centralization. This kind of tool puts your secrets in a secret place, wraps that with encryption and access control, and provides fine-grained audit logging so you know who's doing what, when and only give access to people and applications that need it.
You should also encourage your teams to continuously reference the twelve-factor pattern to reduce application dependencies on credentials from external services. You can also consider using dynamic secrets: a dynamic secret is generated on demand and is unique to a client, instead of a static secret, which is defined ahead of time and shared. Dynamic secrets help reduce the blast damage of any leaked secrets or compromised systems because every authenticated entity will have a unique set of credentials.
Your secrets hierarchy design should account for secrets isolation per application, environment and a fail-proof revocation of secrets when required. To further strengthen the secrets structure, access policies and role-based mappings need to be built to support emergencies by making them version-controlled and automated.
Depending on your platform choices, you can take a look at Docker Secrets, AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, and CyberArk Conjur for secrets management.
It’s important that a secrets management tool has comprehensive and accessible APIs and integrations to other security tools such as machine identity management tools like those offered by Venafi. The Venafi and HashiCorp Vault integration allows DevOps teams to obtain TLS keys and digital certificates using the Machine Identity Management service operated by the organization’s security team: that’s DevSecOps in action.
Find out more about how Venafi and HashiCorp together make it faster and safer for DevOps teams to generate external certificates for their applications here and provide much needed policy controls and visibility for security teams.
Machine Identity Security Architecture
Related posts