Infrastructure Manager is GA with support available in gcloud. Infrastructure Manager release notes | Google Cloud
At Jetstack Consult, we continue to see our clients embrace FleetOps to manage their infrastructure and platforms at scale in Google Cloud. Whilst there are different patterns, tooling and services for managing infrastructure as we’ve seen with Config Connector and Config Controller, many users are still dependent on Terraform to bootstrap cloud organizations and environments.
This article discusses Infrastructure Manager, a managed solution by Google Cloud for executing and orchestrating Terraform deployments in Google Cloud environments. Infrastructure Manager, as of September 2023, provides a native Terraform experience within Google Cloud, simplifying the management of infrastructure at scale. It eliminates the need for users to host and manage Terraform runners and ensures secure and versioned state management.
Typically, when applying Terraform against Google Cloud environments users will opt for Cloud Build or GitHub Actions as an executor. This isn’t a first-class experience, meaning these may be self-managed runners hosted within a Google Cloud project, however, the invocation and lifecycle of Terraform are wrapped in user-owned workflows, with administrators responsible for persisting state in a preconfigured backend.
With Infrastructure Manager, users can simply reference their HCL and variables with execution and state management administered by Google Cloud. This removes the overhead for users to host and manage runners, as well as ensuring state files are sufficiently secure and versioned.
When creating Google Cloud Organizations from scratch with Terraform, as seen in the terraform-example-foundation, or for setting up GKE clusters from which to deploy further cloud resources with Config Connector, Infrastructure Manager represents a no-maintenance, low-ops proposition for Terraform lifecycle management in Google Cloud.
Let’s take a look at how to set up and use Infrastructure Manager for a standard GKE deployment. This involves setting up the necessary IAM permissions, creating an Infrastructure Manager deployment to apply the Terraform, and inspecting how the service executes the job using Cloud Build.
IAM your manager
Firstly, Infrastructure Manager (IM) requires a Google Service Account with sufficient Cloud IAM permissions to create IM deployments and the cloud resources being defined in HCL. The Infrastructure Manager Service Agent IAM role provides access for the service account to manage the Cloud Storage Bucket that stores the Terraform state file, as well as updating the state of a deployment.
export PROJECT_ID=jetstack-paul
export GSA=sa-infra-mgr
gcloud services enable config.googleapis.com
gcloud iam service-accounts create ${GSA}
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
--member="serviceAccount:${GSA}@${PROJECT_ID}.iam.gserviceaccount.com" \
--role=roles/config.agent \
--condition=None
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
--member=serviceAccount:${GSA}@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/compute.networkAdmin \
--condition=None
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
--member=serviceAccount:${GSA}@${PROJECT_ID}.iam.gserviceaccount.com \
--role=roles/container.admin \
--condition=NoneTerraform as a service
The only current method for interacting with Infrastructure Manager is through gcloud, with a few operations available to users for applying Terraform through the service.
Infrastructure Manager can source HCL from a public Git repository, a Cloud Storage Bucket or from a local machine. This can be existing Terraform configuration or publicly available modules such as the Terraform Blueprints for Google Cloud.
At the time of writing (21 Sept 2023), there is no way of previewing Infrastructure Manager changes between revisions using gcloud as would normally be seen in a terraform plan.
The visibility of the Terraform actuation through gcloud is currently limited, with Terraform output streamed to a file in Cloud Storage. However, when an Infrastructure Manager deployment is applied, a Cloud Build Job is created to perform the execution of Terraform.
This Job is invoking Terraform (more on how that’s done below), which performs a terraform init and terraform validate on the HCL before running terraform apply|destroy without any approval (presumably applying with -auto-approve flag). This Job can be followed in the console and shows logs of the job as it’s being executed.
Taking a look at running Infrastructure Manager with gcloud, HCL from Git can be applied directly using the aforementioned Service Account, as well as tfvars passed inline or from a file.
$ tree terraform/
terraform/
├── main.tf
├── variables.tf
└── versions.tf
$ gcloud infra-manager deployments apply \
projects/${PROJECT_ID}/locations/us-central1/deployments/fleetops \
--service-account=projects/${PROJECT_ID}/serviceAccounts/${GSA}@${PROJECT_ID}.iam.gserviceaccount.com \
--git-source-repo=https://github.com/paulwilljones/fleetops-infra-manager \
--git-source-directory=terraform \
--git-source-ref=develop \
--input-values=resource_prefix=fleetops,region=europe-west2,zone=europe-west2-a
Creating the deployment... logs=gs://993897508389-us-central1-blueprint-config/fleetops/r-0/logs ...done.With the deployment applied, it can be described to show the revisions and input values.
$ gcloud infra-manager deployments describe fleetops --location us-central1
createTime: '2023-09-22T09:28:44.161186963Z'
latestRevision: projects/jetstack-paul/locations/us-central1/deployments/fleetops/revisions/r-0
lockState: UNLOCKED
name: projects/jetstack-paul/locations/us-central1/deployments/fleetops
serviceAccount: projects/jetstack-paul/serviceAccounts/gsa-infra-mgr@jetstack-paul.iam.gserviceaccount.com
state: ACTIVE
stateDetail: revision "projects/jetstack-paul/locations/us-central1/deployments/fleetops/revisions/r-0"
applied
terraformBlueprint:
gitSource:
directory: terraform
ref: develop
repo: https://github.com/paulwilljones/fleetops-infra-manager
inputValues:
master_authorized_range:
inputValue: X.X.X.X/32
region:
inputValue: europe-west2
resource_prefix:
inputValue: fleetops
zone:
inputValue: europe-west2-a
updateTime: '2023-09-22T09:36:16.888929378Z'The revisions for a deployment can be listed and described, with resources within a specific revision under management listed and described.
$ gcloud infra-manager revisions list --deployment fleetops --location us-central1
NAME STATE CREATE_TIME UPDATE_TIME
r-0 APPLIED 2023-09-22T09:28:44.838813325Z 2023-09-22T09:36:14.398026538Z
$ gcloud infra-manager revisions describe r-0 --deployment fleetops --location us-central1
action: CREATE
applyResults:
artifacts: gs://993897508389-us-central1-blueprint-config/fleetops/r-0/apply_results/artifacts
content: gs://993897508389-us-central1-blueprint-config/fleetops/r-0/apply_results/content
build: 1c4c334e-e913-45be-a4d2-a273100ad97d
createTime: '2023-09-22T09:28:44.838813325Z'
logs: gs://993897508389-us-central1-blueprint-config/fleetops/r-0/logs
name: projects/jetstack-paul/locations/us-central1/deployments/fleetops/revisions/r-0
serviceAccount: projects/jetstack-paul/serviceAccounts/gsa-infra-mgr@jetstack-paul.iam.gserviceaccount.com
state: APPLIED
stateDetail: revision applied
terraformBlueprint:
gitSource:
directory: terraform
ref: develop
repo: https://github.com/paulwilljones/fleetops-infra-manager
inputValues:
master_authorized_range:
inputValue: X.X.X.X/32
region:
inputValue: europe-west2
resource_prefix:
inputValue: fleetops
zone:
inputValue: europe-west2-a
updateTime: '2023-09-22T09:36:14.398026538Z'
$ gcloud infra-manager resources list --deployment fleetops --location us-central1 --revision r-0
NAME STATE
compute-network-wjy9yoiy RECONCILED
compute-subnetwork-a4znqjl9 RECONCILED
compute-subnetwork-ew4bo9a8 RECONCILED
container-cluster-lusowmor RECONCILED
$ gcloud infra-manager resources describe container-cluster-lusowmor --deployment fleetops --location us-central1 --revision r-0
caiAssets:
container.googleapis.com/Cluster:
fullResourceName: //container.googleapis.com/projects/jetstack-paul/locations/europe-west2-a/clusters/gke-fleetops-kcc-euw2
intent: CREATE
name: projects/jetstack-paul/locations/us-central1/deployments/fleetops/revisions/r-0/resources/container-cluster-lusowmor
state: RECONCILED
terraformInfo:
address: google_container_cluster.cluster
id: projects/jetstack-paul/locations/europe-west2-a/clusters/gke-fleetops-kcc-euw2
type: google_container_cluster
With the state file persisted in Cloud Storage, it can be locked/unlocked, exported for local inspection, and imported back into the bucket using a signed URL.
As of now, only Terraform version 1.2.3 is supported, therefore all providers and imported modules must be compatible with this version. Also, defining backend blocks or provisioners is not supported.
Look ma, no plans
As mentioned, with Infrastructure Manager only having gcloud alpha support, Terraform can currently only be applied and not planned.
However, looking closer at how Infrastructure Manager is executing Terraform in Cloud Build shows what is happening under the hood and gives us a glimpse of what the service will look like going forward.
The Cloud Build Job is running a Docker container to execute the process of sourcing the HCL, initialising Terraform, performing actuation and persisting the state in GCS.
As the Docker Image being used is publicly available, it can be pulled for further inspection.
$ docker run -it --rm gcr.io/cloud-config-sdk/config-sdk-tf:v0.0.96 ls /usr/local/bin
config-sdk go-gcloud terraform
$ docker run -it --rm gcr.io/cloud-config-sdk/config-sdk-tf:v0.0.96 /usr/local/bin/config-sdk
CLI tool for Blueprints Controller
Usage:
config-sdk [command]
Available Commands:
...
tf Preview or apply Terraform blueprints
...
$ docker run -it --rm gcr.io/cloud-config-sdk/config-sdk-tf:v0.0.96 /usr/local/bin/config-sdk tf --help
Preview or apply Terraform blueprints
Usage:
config-sdk tf [command]
Aliases:
tf, terraform
Available Commands:
apply applies a terraform blueprint
destroy destroy a terraform blueprint
plan plans a terraform blueprint
Flags:
--backend backend type The backend TF should use for state storage. One of [IM FS GCS] (default GCS)
--backend-bucket string The GCS bucket to use as remote backend. Example: "gs://tf/state"
--deployment string Deployment name of form //config.googleapis.com/apiVersion/projects/{project}/locations/{location}/deployments/{deployment}.
--inputs string A JSON object with keys as input names and values as input values.
--source-git string The Git source to retrieve blueprint from. HTTPS or SSH are supported. Example: "https://github.com/terraform-google-modules/terraform-google-project-factory"
--source-git-ref string Git branch or tag to fetch. Example: "v1.0.0"
--source-subdir string Sub directory within a bucket or repo if blueprint is not at root. Example: "modules/project_services"Seeing that the config-sdk binary has a tf plan subcommand suggests that this capability will soon come to gcloud. Running the Docker Image locally to use the tf plan command with the same flags as the gcloud infra-manager deployments apply gives the expected terraform plan output for the revision.
$ docker run -it --rm -v ~/.config/gcloud/application_default_credentials.json:/root/.config/gcloud/application_default_credentials.json gcr.io/cloud-config-sdk/config-sdk-tf:v0.0.96 /usr/local/bin/config-sdk tf plan --backend im --backend-bucket gs:/
/993897508389-us-central1-blueprint-config/fleetops/state --deployment //config.googleapis.com/v1alpha2/projects/jetstack-paul/locations/us-central1/deployments/fleetops --inputs '{"resource_prefix":"fleetops","region":"europe-west2","zone":"europe-west2-a","master_authorized_range":"X.X.X.X/32"}' --output-bucket gs://993897508389-us-central1-blueprint-config/fleetops/r-999/apply_results --source-git https://github.com/paulwilljones/fleetops-infra-manager --source-subdir terraform/
INFO[0001] Logging to GCS object: gs://993897508389-us-central1-blueprint-config/artifacts/log.json
INFO[0001] fetching blueprint from git::https://github.com/paulwilljones/fleetops-infra-manager.git
INFO[0003] Running terraform init in /workspace/content/terraform
Initializing modules...
...
module.gcp-network.module.vpc.google_compute_network.network: Plan to create
module.gcp-network.module.vpc.google_compute_network.network: Plan to create
module.gcp-network.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/sb-fleetops-master-euw2"]: Plan to create
module.gcp-network.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/sb-fleetops-master-euw2"]: Plan to create
module.gcp-network.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/sb-fleetops-kcc-euw2"]: Plan to create
module.gcp-network.module.subnets.google_compute_subnetwork.subnetwork["europe-west2/sb-fleetops-kcc-euw2"]: Plan to create
google_container_cluster.cluster: Plan to create
google_container_cluster.cluster: Plan to create
...
Plan: 8 to add, 0 to change, 0 to destroy.
INFO[0107] Running terraform show in /workspace/content/terraform on /tmp/tfplan-1618541012/plan.outBring your own providers
Whilst Infrastructure Manager executes the Terraform workflow in Google Cloud, there are no restrictions as to what Providers or resources (apart from Provisioners) can be included in HCL.
Existing Terraform codebases that create resources across other cloud providers or systems can be executed by Infrastructure Manager. Providers can be configured by integrating with other Google Cloud services like Secrets Manager to retrieve credentials.
data "google_secret_manager_secret_version" "aws_access_key" {
secret = "aws-access-key"
project = data.google_project.project.project_id
}
data "google_secret_manager_secret_version" "aws_secret_key" {
secret = "aws-secret-key"
project = data.google_project.project.project_id
}
provider "aws" {
region = "us-east-1"
access_key = data.google_secret_manager_secret_version.aws_access_key.secret_data
secret_key = data.google_secret_manager_secret_version.aws_secret_key.secret_data
}
resource "aws_eks_cluster" "example" {
name = local.cluster_name
role_arn = data.aws_iam_role.cluster.arn
vpc_config {
subnet_ids = [aws_subnet.example1.id, aws_subnet.example2.id]
}
}This allows Infrastructure Manager to adopt existing Terraform codebases and be the centralised actuator for all infrastructure-as-code deployments.
Lastly, there is no current support for:
- Terraform Workspaces
- Terraform wrappers (i.e., Terragrunt)
- No environment variables or workspace variables
Update in-place
Infrastructure Manager offers a managed solution for hosting and running Terraform workflows in Google Cloud. It’s early in the service’s roadmap, but the core capability and value-add of managing Terraform deployments on Google Cloud is clear, with more features to come going into Beta and GA support.
The proposition reduces the overhead of managing Terraform execution, removing the need for hosting runners as well as securing and administering state management.
There is a growing appetite for teams to adopt managed services in order to reduce the time spent managing tooling and instead focus on building solutions for their users.
Infrastructure Manager provides a no-ops way of running Terraform in Google Cloud to minimise operational toil and responsibility.
Importantly, with Infrastructure Manager being a Google Cloud API, its usage and invocation can be governed by Cloud IAM permissions. Instead of self-managing the access to Terraform execution and state files, users and service accounts can simply be granted the necessary Cloud IAM permissions in order to launch a terraform apply.
With the creation of infrastructure using Terraform now possible through a gcloud command, it opens up the possibilities for bootstrapping environments and chaining together patterns for provisioning platforms on GKE.
See this post for how to use the newly provisioned GKE cluster with Config Connector and Anthos Config Management enabled to manage Fleets at scale from a centralised management plane.
Next Steps
To wrap up this FleetOps discussion on Infrastructure Manager and its benefits for Terraform deployments in Google Cloud, we invite you to explore the possibilities of streamlining your infrastructure management. At Jetstack Consult, we have a wealth of experience in optimising Google Cloud deployments, with field expertise to assist you in leveraging Infrastructure Manager and other tools to enhance your cloud infrastructure. Whether you have questions, need guidance, or want to explore how our expertise can benefit your organisation, don't hesitate to reach out. Contact Jetstack Consult and let’s start a conversation about your Google Cloud deployment needs, and how we can help you achieve your cloud infrastructure goals.
Related posts