GCP Goodies Part 1 — Google Deployment Manager Basics
Once you start setting up cloud infrastructure consisting of more than a single server and a database you will realize the importance of Infrastructure as Code (IaC) solutions such as Terraform, Puppet, etc, among many others. Recently, within a project I work on, we have moved all our infra to Google and its cloud offering, which made me realize that Google has its own tool to manage infrastructure in elegant way called Deployment Manager, and in some cases it can be even better than the Terraform we have normally used. This post will not, in any way, try to persuade you to move from Terraform to Google Deployment Manager but the DM has some pros of which every cloud developer should know.
Google Deployment Manager pros:
- You can use python and execute your custom logic within the templates
- No need to manage state by yourself as its a hosted solution
- Works with most (alpha, beta) GCP API calls
- Can be used to manage Kubernetes state
Terraform pros:
- you have the added benefit of knowing precisely what has changed since you’re last deployment and what will be changed during the next one
- It’s multi platform
- Widely used
This is of course kind of “flame” discussion, both tools have pros and cons. The main advantage of DM versus Terraform is the ability to create templates in Python/Jinja where you can add custom logic to your infrastructure as code.
You can read more about Deployment Manager here.
What is cool about Deployment Manager is that you don’t need almost any tools installed to get up and running. We will execute all our commands from the Google Cloud Shell and available there code editor.
Go to Google Console, select your project and click on the cloud shell icon to enter the cloud shell terminal window.
Alternatively, you can change your project while you are in a Cloud Shell already with a command:
gcloud config set project softwaremill-playground
Once in a cloud shell, clone the repository with Deployment Manager samples,
git clone https://github.com/GoogleCloudPlatform/deployment-manager-samples.git
Simple Configuration
Let’s dive into code, first of all we create a very simple configuration file in YAML, where we set up 2 virtual machines in Google Compute Engine space, with some details like source image, machine type etc. This is a pretty basic config (taken from the google deployment manager examples) to show you how easy it can be. Of course simple yaml file like this lacks a lot of features I want to discuss here but it’s just for starters.
To run it, navigate to step-by-step-guide/step2_create_a_configuration , change all occurrences of MY_PROJECT variable to your project name within the file itself and run DM create with the deployment name of your choice ( two-vms-1 in the example below):
Navigate to Compute Engine -> VM Instances to see the changes your recent deployment created.
To delete the resources, delete the deployment manager itself with:
References
Now, let’s say you want to have a common network for those two instances we have defined earlier, we can define the network as a third resource and then reference that network within configuration for each separate virtual machine:
First Template
You don’t need to specify every attribute for your needed resources directly in the yaml file, within each project the need arises quickly to extract some properties which can change to external file and have a template which will read that properties and build the final configuration we want to deploy to the cloud. Deployment Manager templates come to the rescue here. At the moment, you have two options to choose from, you can write your templates in Jinja or Python. I will show you configurations for Python only, the choice is yours of course, some people consider Jinja to be more readable, some people (like me) already know Python so it is a no-brainer.
First VM:
Second VM:
Config:
To run it, navigate to step-by-step-guide\step5_create_a_template\python, change all occurrences of MY_PROJECT to your project name, and execute the following:
In this case, except for the two virtual machines created like in the previous example, you will see a custom network configuration in VPC Networks section of the Console:
and in the VM instance details you will see that VMs got attached to it:
Multiple Templates
You can of course, use many templates and join them together within imports section of your final YAML configuration file.
Please have a look into the examples provided by google for the full source code.
Config with many templates looks the same as the one in previous example:
The ‘joining’ template is the one called compute-engine-template.py and contains all the templates gathered within single resources section:
Environment Variables
Passing environment variables to your templates is pretty easy, you reference them with like a Map attributes eg:
Setting up GKE Cluster
Previous examples were all nice and easy but Deployment Manager capabilities don’t end here. We have a whole new world ahead of use with the ability to use Type Providers.
Citing Google on this:
Deployment Manager offers the ability to register a third-party API with the Deployment Manager service. After registering an API as a type provider with Deployment Manager, you can use Deployment Manager to deploy resources from the API as types in your configuration.
What it means is that we can use third party APIs in our Deployment Manager code to set up additional things (like in case of Kubernetes — manage the kubernetes cluster)
In this post we will play around with the Kubernetes cluster deployment only but I’m pretty sure that we will dive into Type provider for Kubernetes in more detail in the future.
Navigate to:
cd deploymentmanager-samples/examples/v2/gke/python
Execute gcloud compute zones list to get the list of zones available and pick one where you want to provision your GKE cluster.
NAME=softwaremill-gke-cluster
ZONE=us-west2-a
Change the name of the cluster (must be unique) and zone to suit your needs.
Execute the following command:
If you see something similar to the message below:
API [deploymentmanager.googleapis.com] not enabled on project
[42043567059]. Would you like to enable and retry (this will take a
few minutes)? (y/N)?
Engine API has not been used in project 42043567059 before or it is disabled.
Enable it by visiting https://console.cloud.google.com/apis/api/container.googleapis.com/overview?project=42043567059
Navigate to the given link and enable the API for the deployment manager to do its work.
Once the command finishes executing, go to Console, enter the menu on the left side and pick ‘Kubernetes Engine’ -> ‘Clusters’. You should see your newly created cluster up and running.
On top of that you will be able to see the deployment you have created with Deployment Manager, navigate to ‘Deployment Manager’ -> ‘Deployments’
Updating the deployment
Open up the cloud shell file editor:
And navigate to the catalog you had open in a cloud shell itself
deploymentmanager-samples/examples/v2/gke/python
Open up cluster.yaml file and change name, zone property and initial node count (notice I have set zone to the same value I’ve used before — this way you don’t need to specify this on the command line itself) only the initialNodeCount was added.
Execute Deployment Manager update in preview mode to see if all is good:
gcloud deployment-manager deployments update ${NAME} --config cluster.yaml --preview
If all looks good you can execute the same command again without config and preview parameters specified. Deployment Manager uses your last preview to perform the update.
The cluster got recreated.
Notice the changed value for Cluster size property on Google Console UI.
Once you finish playing around with your cluster you can easily delete it:
gcloud deployment-manager deployments delete ${NAME}
This command deletes the deployment as well as the cluster it created. Navigate to Kubernetes Engine -> Clusters to make sure everything was cleaned up properly.
GCP Goodies is intended to be a series of blog posts about multiple services available for developers on Google Cloud. Watch this space for more on Google Deployment Manager (especially Type Providers) and other Google Cloud offerings to ease out day to day software development.
