Google Cloud Landing Zone with Terraform and Cloud Foundation Fabric FAST - Part 2

on April 10, 2023

Introduction

Here I provide step-by-step guidance for setting up a new Landing Zone on Google Cloud, using Google’s open source Fabric FAST, which is part of their Cloud Foundation Fabric.

This is Part 2. If you haven’t already done so, you should make sure you’ve completed the steps in Part 1.

About Part 2

If you’re here, then you’ve already set up your Cloud Identity account, your organisation, your groups and users, and you’ve cloned the CFF GitHub repo to your local machine.

Part 1 involved quite a few manual setup tasks. We were following the initial steps from the Google Cloud setup checklist, using the Cloud foundation guided steps. We have completed Steps 1 and 2 from the Checklist:

But from here onwards, we will be using Fabric FAST to complete the all the remaining activities using Infrastructure-as-Code. So, to be clear: we will not be contuining with this guided checklist:

In this part, we will:

Run FAST stages/0-bootstrap – to configure automation, billing, and log export projects, custom roles, service accounts, organisation-level logging, and workload identity federation support.
Migrate Terraform state to a GCS bucket.
Run FAST stages/1-resman – to provide “shared capability” folders like networking and security, and to apply a set of top-down organisation-level security policies.
Optionally run FAST stages-multitenant/0-bootstrap-tenant – to provide optional top-level tenant folders.

FAST Overview

Within the CFF repo, you will find a folder called fast, which contains Fabric FAST.

Fabric FAST provides an automated infrastructure-as-code approach for creating a production-ready landing zone for an organisation. Although customisable, it provides a way to deploy a landing zone with “battle-tested” default configuration options. Furthermore, the intent is to delegate as much responsibility as possible to the tenants who will consume the landing zone. Much of this delegation is in the form of factories, which allow tenants to spin up their own tenant folders and tenant projects and resources, simply by supplying a configuration file.

Google describes FAST as “an ideal blueprint for organizations of all sizes, ranging from startups to the largest companies.”

Stages

Fabric FAST is split into multiple stages. The stages are layered, meaning that a given stage is dependent on having run the preceding stages.

This folder in the repo gives details about the FAST stages. But I’ll summarise them here:

Bootstrap – provides foundational organisation-level configuration, to enable steps that depend on broad administrative permissions; prepares the prerequisites needed to enable automation in this and future stages.
Resource management – creates the base resource hierarchy (folders) and the automation resources required later, in order to delegate deployment of each part of the hierarchy to separate stages.
Networking – provides centralised networking resources, using a hub-and-spoke design.
Security – implements security configuration, adhering to best practices.
Project Factory – creates a YAML-based factory that allows projects to be created for any given tenant (e.g. an application or team). This should be run once per environment, e.g. for dev, qa, staging, prod.
Data platform – an optional stage intended for the deployment of components that would be needed to build a data platform, such as BigQuery, Pub/Sub, Dataflow, and Cloud Composer.
GKE Multitenant – an optional stage for building a shared GKE environment that can host multiple tenant applications

We can look at the stages more closely by opening the stages folder in your cloned FAST repo.

Each FAST stage has its own folder in the repo. Each of these folders represents a Terraform root module. I.e. a folder where we can run terraform commands to execute a given stage in the overall process.

Multitenant Top Level

One useful feature of FAST is to be able to bootstrap such that the top level of the FAST resource hierarchy is a folder under the organisation, rather than in the organisation itself. This is useful if you potentially want more than one FAST hierarchy deployed into your organisation. Or, in my case, where I have a single organisation, and I want to keep FAST experimentation separate from any other projects I might deploy.

If you want to do this, you’ll need to additionally use the stages-multitenant stages, in the sequence shown in the diagram above.

Pre-Reqs

Most of the pre-reqs will have been completed in Part 1. However, you will need an environment running all of: Google Gcloud CLI, Terraform, and git.

Google Gcloud CLI (formerly known as Google Gcloud SDK) is a set of command line tools for interacting with Google Cloud from the command line, including gcloud, gsutil, bq, and kubectl.

One option is to use Google Cloud Shell. As a reminder: Cloud Shell is a command line interface hosted on an ephemeral Debian-based Google Compute Engine instance, accessible from the Google Cloud Console. It has Google Gcloud CLI pre-installed, as well as other tools like Terraform and git. Furthermore, Cloud Shell also comes with a built-in code editor. So if you’re using Cloud Shell, you won’t have to install any development environment locally.

One downside: Cloud Shell has a usage quota limit of 50 hours per week. If there’s a chance you’re going to exceed 50 hours, you might want to use a different environment… Or request a quota increase from Google.

Alternatively, you can install the Google Gcloud CLI locally, by following the instructions here. If you’re on Windows, I’d recommend installing it using winget. It’s so easy!

winget install Google.CloudSDK

After installing, restart your current local CLI and then verify your installation, e.g.

One more consideration: many of the FAST instructions assume you are running command in a Linux environment. If you’re on Windows but you have more familiarity with Linux, you may want to set up a Linux environment within Windows. I would recommend using Windows Subsystem for Linux (WSL) with Ubuntu, and then installing the gcloud CLI using the instructions here. After installing, you can check your gcloud CLI version:

Stage 0: Bootstrap

FAST starts with the 0-boostrap stage. Open this folder in your cloned repo. You’ll see a detailed description of this stage. In your development environment, make sure you are in the fast/stages/0-bootstrap folder.

Overview

Here is a quick overview of what this stage achieves:

Creates custom roles that allows setting IAM policies at the organisation level, to allow the Resource Management service account to grant a specific set of roles.
Creates a service account for the next stage.
Creates an automation project for running subsequent FAST tasks.
Creates GCS buckets to act as the remote backed for Terraform state, and for storing FAST stage output files.
Creates a billing export project.
Creates a log export project.
Sets up organisation-level logging: org-level log sinks are created, to ensure a proper audit trail right from the start. By default, FAST provides log filters to capture Cloud Audit Logs and VPC Service Controls violations into a Bigquery dataset in a newly created top-level audit project.
Enables Workload Identity Federation support with external providers. This allows non-Google external identies (such as Active Directory or OIDC identity providers like Ping Identity or Okta) to be used to access Google Cloud resources, without having to obtain and share a service account key.

Steps

Login

Launch a Cloud Console browser session, using your Org Admin user. If you want to use the Cloud Shell, simply launch it from the console.

If you’d prefer to use your local shell:

# login to the gcloud CLI as the Org Admin user
gcloud auth login

# set the gcloud CLI default application credentials to the Org Admin user
gcloud auth application-default login

You will be given a link to open in your browser. Paste in the link, and when prompted to choose an account, make sure you select your Org Admin account.

Google will give you a verification code, which you can now paste back into your gcloud CLI command.

Grant Roles

Now we will self-grant required roles to our Org Admin account:

Billing Account Administrator (roles/billing.admin) either on the organization or the billing account (see the following section for details)
Logging Admin (roles/logging.admin)
Organization Role Administrator (roles/iam.organizationRoleAdmin)
Organization Administrator (roles/resourcemanager.organizationAdmin)
Project Creator (roles/resourcemanager.projectCreator)

# set variable for current logged in user
export FAST_BU=$(gcloud config list --format 'value(core.account)')

# find and set your org id
gcloud organizations list
export FAST_ORG_ID=<your org ID>

# set needed roles
export FAST_ROLES="roles/billing.admin roles/logging.admin \
  roles/iam.organizationRoleAdmin \
  roles/resourcemanager.projectCreator"

for role in $FAST_ROLES; do
  gcloud organizations add-iam-policy-binding $FAST_ORG_ID \
    --member user:$FAST_BU --role $role
done

# find your billing account ID
gcloud alpha billing accounts list

export FAST_BILLING_ACCOUNT_ID=<your billing account id>
gcloud beta billing accounts add-iam-policy-binding $FAST_BILLING_ACCOUNT_ID \
  --member user:$FAST_BU --role roles/billing.admin

Create Terraform Variables

Create a terraform.tfvars file, in the 0-boostrap folder. This will be used to supply the stage with the necessary variables for your environment. Note that by default, this terraform.tfvars file will not be under git source control, since it is excluded in the .gitignore file. This is to prevent you from checking potentially sensitive environment-specific information. (If you do wish to source control your variables, consider making them as sensitive, and storing the files themselves as secrets. I’m not going to cover this here.)

Note: project IDs in Google Cloud need to be globally unique. So, think carefully about the prefix you want to use. The first prefix I tried was “acme”, which – surprise, surprise – did not result in globally unique project IDs!!

# use `gcloud beta billing accounts list`
# if you have too many accounts, check the Cloud Console :)
billing_account = {
 id              = "012345-67890A-BCDEF0"
}

# use `gcloud organizations list`
organization = {
 domain      = "example.org"
 id          = 1234567890
 customer_id = "C000001"
}

outputs_location = "~/fast-config"

# use something unique and no longer than 9 characters
prefix = "acme"

Run the Terraform!

# Setup application default credentials (ADC)
gcloud auth application-default login

terraform init

# optional - if you want to see what the apply command will do
terraform plan

# apply the configuration
terraform apply -var bootstrap_user=$(gcloud config list --format 'value(core.account)')

It will take a couple of minutes to run.

Afterwards, you’ll see three new projects have been created:

And new log sinks will be visible:

And if we open the billing project, we can see a BigQuery billing export dataset has been created:

In the iac project, we can see the new GCS buckets that have been created:

Cool, right?

Output Variables

The 0-bootstrap stage creates a number of output variables. By default, it writes them to the folder ~/fast-config. This can be changed in the .tfvars file we supplied earlier.

Switching State to the Remote Backend

When we first run terraform init, Terraform stores its state in a local file. However, it is a good idea to move this state into a so-called remote backend. I.e. a storage location that is accessible over a network and can be used collaboratively. This page describes the general process of migrating Terraform state to a Google Cloud Storage bucket. However, FAST facilitates this process for us. The 0-bootstrap guidance tells us the steps we need to take, to migrate the TF state:

# to copy the FAST output files from the local fast-config directory:
../../stage-links.sh ~/fast-config

# alternatively, we can copy the FAST output files from the GCS bucket:
export PREFIX=<your prefix>
../../stage-links.sh gs://$PREFIX-prod-iac-core-outputs-0

# If you have any "bad interpreter" errors, you may have issues with your end-of-line chars
# If so, fix with this line, then try again
sed -i -e 's/\r$//' ../../stage-links.sh

The command will return a subsequent ln command, which is specific for this stage. Running this line creates a symbolic link from the newly created 0-boostrap-providers.tf file, to the current working directory. Again, this file will not be checked-in to source control, as it is excluded in .gitignore.

Now we can migrate the state to the GCS bucket:

terraform init -migrate-state
terraform apply

Collaborative Working

Having now migrated the state, if you want to continue the FAST process from another machine, you can! To resume TF steps on another machine you will need to:

# Ensure pre-reqs: we have installed gcloud CLI, terraform, git
# Ensure terraform.tfvars matches what was set on the other machine
# Login with gcloud CLI, and set ADC
gcloud auth login
gcloud auth application-default login

# Change to: fast/stages/0-bootstrap folder
# Copy the external backend state providers.tf file to the local directory
gsutil cp gs://jtg-acme-prod-iac-core-outputs-0/providers/0-bootstrap-providers.tf ./

# Initialise Terraform; the state will be pulled from the GCS bucket
terraform init
terraform apply

Stage 1 – Resource Hiearchy

FAST continues with the 1-resman stage. Open this folder in your cloned repo. You’ll see a detailed description of this stage. In your development environment, make sure you are in the fast/stages/1-resman folder.

Overview

Here is a quick overview of what this stage achieves:

Create the top-level hierarchy of folders and associated resources.
Sets organisation policies, and any exception required on specific folders.

The diagram below shows the resource hiearchy that is created.

Notes:

Separate folders are created for “shared” capabities: Networking and Security.
A separate Sandbox folder is created. The intent of this folder is for experimentation with resources that are not subject to the same levels of IaC-enforced control as the rest of the hierarchy.
A top-level Teams folder, which contains separate folders for each “application/team” tenant.
Subsequent shared capabilities – e.g. a multitenant GKE cluster – could be added without compromising this design.
This stage is a pre-requisite for provisioning top-level tenants. (See below.)

Which Organisation Policies?

If you like in the folder stages/1-resman/data/org-policies, you can view the various policies that are set by FAST. These include, but are not limited to:

Enforce compute.requireOsLogin
Enforce compute.skipDefaultNetworkCreation
Deny compute.vmExternalIpAccess
Enforce iam.disableServiceAccountKeyCreation
Enforce sql.restrictPublicIp
Enforce storage.uniformBucketLevelAccess

Of course, you are free to edit these files, to configure the org policies you want to apply.

Steps

Add a terraform.tfvars file. E.g.

team_folders = {
  team-foo = {
    descriptive_name = "Team Foo"
    group_iam = {
      "team-foo@just2good.co.uk" = [
        "roles/viewer"
      ]
    }
    impersonation_groups = ["team-foo-admins@just2good.co.uk"]
  }
}

outputs_location = "~/fast-config"

We run the stage-links.sh script to obtain the commands to copy output from the previous stage.

# either
../../stage-links.sh ~/fast-config
# or
../../stage-links.sh gs://$PREFIX-prod-iac-core-outputs-0

As per the instructions, copy the commands and run them.

Now run the Terraform:

# Initialise Terraform; the state will be pulled from the GCS bucket
terraform init
terraform apply

For me, this creates 63 new resources and takes a couple of minutes to run.

You’ll see a bunch of new projects have been created. (Ignore “scratch” in my screenshot.)

Configuring a Tenant “Top Level”: 0-bootstrap-tenant

Now we can optionally proceed to the 0-boostrap-tenant stage. Open the stages-multitenant/0-bootstrap-tenant folder in your cloned repo, and look at the guidance. In your development environment, make sure you are in the fast/stages-multitenant/0-bootstrap-tenant folder.

Remember that we need to have run both the 0-bootstrap and 1-resman stages, before running this stage.

Overview

Here we create a top-level tenant folder, under the organisation node. This stage creates service accounts for all tenant stages, such that Billing and Organization Policy Admin bindings can be set, leveraging permissions of the org-level resman service account which is used to run this stage. This avoids the need to grant broad scoped permissions on the organisation to tenant-level service accounts, thus decoupling the tenant from the organisation.

Steps

# either
../../stage-links.sh ~/fast-config
# or
../../stage-links.sh gs://$PREFIX-prod-iac-core-outputs-0

As per the guidance:

Copy all the commands, paste them and run them.
Edit the 0-bootstrap-tenant-providers.tf file, and supply the name of your top-level tenant, as the variable “prefix”.

terraform {
  backend "gcs" {
    bucket                      = "jtg-acme-prod-iac-core-resman-0"
    impersonate_service_account = "jtg-acme-prod-resman-0@jtg-acme-prod-iac-core-0.iam.gserviceaccount.com"
    # remove the newline between quotes and set the tenant name as prefix
    prefix = "dazbo-lz"
  }
}
provider "google" {
  impersonate_service_account = "jtg-acme-prod-resman-0@jtg-acme-prod-iac-core-0.iam.gserviceaccount.com"
}
provider "google-beta" {
  impersonate_service_account = "jtg-acme-prod-resman-0@jtg-acme-prod-iac-core-0.iam.gserviceaccount.com"
}

# end provider.tf for bootstrap-tenant

Now provide a terraform.tfvars file, and provide the required config:

tenant_config = {
  # used for the top-level folder name
  descriptive_name = "Dazbo Tenant A"
  # tenant-specific groups, only the admin group is required
  # the organization domain is automatically added after the group name
  groups = {
    gcp-admins          = "tnta-admins"
    # gcp-devops          = "tnta-devops"
    # gcp-network-admins  = "tnta-networking"
    # gcp-security-admins = "tnta-security"
  }
  # the 3 or 4 letter acronym or abbreviation used in resource names
  short_name = "tnta"
  # optional CI/CD configuration, refer to the org-level stages for information 
  # cicd = {
  #   branch            = null
  #   identity_provider = "foo-provider"
  #   name              = "myorg/tnta-bootstrap"
  #   type              = "github"
  # }
  # optional group-level IAM bindings to add to the top-level folder
  # group_iam = {
  #   tnta-support = ["roles/viewer"]
  # }
  # optional IAM bindings to add to the top-level folder
  # iam       = {
  #   "roles/logging.admin" = [
  #     "serviceAccount:foo@myprj.iam.gserviceaccount.com"
  #   ]
  # }
  # optional location overrides to global locations
  # locations = {
  #   bq      = null
  #   gcs     = null
  #   logging = null
  #   pubsub  = null
  # }
  # optional folder ids for automation and logging project folders, typically
  # added in later stages and entered here once created
  # project_parent_ids = {
  #   automation = "folders/012345678"
  #   logging    = "folders/0123456789"
  # }
}
# tftest skip

outputs_location = "~/fast-config"

One important thing to note is that we’ve defined a new admin Google group called tnta-admins. This group needs to exist before we can apply the Terraform. So, in the Admin Console, add this new group.

Now run Terraform, as usual:

# Initialise Terraform; the state will be pulled from the GCS bucket
terraform init
terraform apply

Finally, we can see that our top-level tenant folder has been created:

Note that this stage also creates staging buckets that are tenant specific:

Configuring a Tenant “Top Level”: 1-resman-tenant

If we ran the stages-multitenant/0-bootstrap-tenant then we should now run stages-multitenant/1-resman-tenant. Switch to this folder in your cloned repo, and look at the guidance. In your development environment, make sure you are in the fast/stages-multitenant/1-resman-tenant folder.

We must run this stage for every top-level tenant we previously created using 0-bootstrap-tenant.

Overview

This stage is decoupled from organization-level stages. It uses a service account and state bucket from the tenant-specific automation project, and its tfvars and provider files are also tenant-specific.

Steps

As with all the other steps, we begin with stage-links.sh to copy provider and output files created by previous stages. But because this stage is also top-level tenant-specific, we must also pass the tenant name to the script. We should use the short_name field that we specified in the previous stage.

Note that the GCS bucket is tenant-specific. So the GS URI is different to previous commands.

export TENANT=<your tenant short name>

# Either
../../stage-links.sh ~/fast-config

# Or
../../stage-links.sh gs://$PREFIX-$TENANT-iac-core-outputs-0

As usual, copy and paste the commands supplied, to execute them.

Then, run Terraform as usual:

terraform init
terraform apply

At this stage, I faced an issue… The original prefix (which we defined all the way back in 0-bootstrap) must be no more than 9 characters long. And the tenancy short_name (which we defined in 0-bootstrap-tenant) must be 3 or 4 characters long. The 1-resman-tenant main.tf defines a prefix variable which must also be no more than 9 characters long. However, the output file stages-multitenant/0-bootstrap-tenant.auto-tfvars.json sets a new prefix variable by concatenating the original prefix with the short_name. And this results in a new prefix variable which is longer than 9 characters, and this fails Terraform validation. To fix this, I changed the stages-multitenant/1-resman-tenant/variables.tf validation rule:

variable "prefix" {
  # tfdoc:variable:source 0-bootstrap
  description = "Prefix used for resources that need unique names. Use 9 characters or less."
  type        = string

  validation {
    condition     = try(length(var.prefix), 0) < 14
    error_message = "Use a maximum of 13 characters for prefix (which incudes tenant short name)."
  }
}

Update: I fixed the above issue in the Cloud-Foundation-Fabric repo itself, so you’ll never face this problem! (That’s the great thing about open source… Anyone can contribute and fix stuff.)

After applying you will see additional “networking” and “security” folders have been created, under the top-level tenant:

Summary

In this part, we have:

Executed FAST stages/0-bootstrap – to configure automation, billing, and log export projects, custom roles, service accounts, organisation-level logging, and workload identity federation support.
Migrated Terraform state to a GCS bucket.
Executed FAST stages/1-resman – to provide “shared capability” folders like networking and security, and applied a set of top-down organisation-level security policies, to ensure we’re secure from the start.
Executed FAST stages-multitenant/0-bootstrap-tenant – to provide optional top-level tenant folders.
Executed FAST stages-multitenant/1-resman-tenant – to create Networking and Security “shared” folders under the top-level tenancy.

We’re now ready to apply the next FAST stages, i.e. “Shared Resources”. This is made up of two stages:

Security
Networking

Watch this space for Part 3!

Useful Links

Categories:

Cloud and Dev IT

Tags:

GCP Google Cloud Landing Zone terraform

Google Cloud Landing Zone with Terraform and Cloud Foundation Fabric FAST – Part 2

Introduction

About Part 2

FAST Overview

Stages

Multitenant Top Level

Pre-Reqs

Stage 0: Bootstrap

Overview

Steps

Login

Grant Roles

Create Terraform Variables

Run the Terraform!

Output Variables

Switching State to the Remote Backend

Collaborative Working

Stage 1 – Resource Hiearchy

Overview

Which Organisation Policies?

Steps

Configuring a Tenant “Top Level”: 0-bootstrap-tenant

Overview

Steps

Configuring a Tenant “Top Level”: 1-resman-tenant

Overview

Steps

Summary

Useful Links

Like this:

Related

No responses yet

Leave a ReplyCancel reply

Google Cloud Landing Zone with Terraform and Cloud Foundation Fabric FAST – Part 2

Introduction

About Part 2

FAST Overview

Stages

Multitenant Top Level

Pre-Reqs

Stage 0: Bootstrap

Overview

Steps

Login

Grant Roles

Create Terraform Variables

Run the Terraform!

Output Variables

Switching State to the Remote Backend

Collaborative Working

Stage 1 – Resource Hiearchy

Overview

Which Organisation Policies?

Steps

Configuring a Tenant “Top Level”: 0-bootstrap-tenant

Overview

Steps

Configuring a Tenant “Top Level”: 1-resman-tenant

Overview

Steps

Summary

Useful Links

Share this:

Like this:

Related

No responses yet

Leave a ReplyCancel reply