Kubernetes Solutions

Managing Deployments Using Kubernetes Engine

Introduction to deployments

Heterogeneous deployments typically involve connecting two or more distinct infrastructure environments or regions to address a specific technical or operational need. Heterogeneous deployments are called “hybrid”, “multi-cloud”, or “public-private”, depending upon the specifics of the deployment. For the purposes of this lab, heterogeneous deployments include those that span regions within a single cloud environment, multiple public cloud environments (multi-cloud), or a combination of on-premises and public cloud environments (hybrid or public-private).

Various business and technical challenges can arise in deployments that are limited to a single environment or region:

Heterogeneous deployments can help address these challenges, but they must be architected using programmatic and deterministic processes and procedures. One-off or ad-hoc deployment procedures can cause deployments or processes to be brittle and intolerant of failures. Ad-hoc processes can lose data or drop traffic. Good deployment processes must be repeatable and use proven approaches for managing provisioning, configuration, and maintenance.

Three common scenarios for heterogeneous deployment are multi-cloud deployments, fronting on-premises data, and continuous integration/continuous delivery (CI/CD) processes.

The following exercises practice some common use cases for heterogeneous deployments, along with well-architected approaches using Kubernetes and other infrastructure resources to accomplish them.

gcloud is the command-line tool for Google Cloud Platform. It comes pre-installed on Cloud Shell and supports tab-completion.

You can list the active account name with this command:

gcloud auth list


Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)

Example output:

Credentialed accounts:
 - google1623327_student@qwiklabs.net

You can list the project ID with this command:

gcloud config list project


project = <project_ID>

Example output:

project = qwiklabs-gcp-44776a13dea667a6

Set zone

Set your working GCP zone by running the following command, substituting the local zone as us-central1-a:

gcloud config set compute/zone us-central1-a

Get sample code for this lab

Get the sample code for creating and running containers and deployments:

git clone https://github.com/googlecodelabs/orchestrate-with-kubernetes.git
cd orchestrate-with-kubernetes/kubernetes

Create a cluster with five n1-standard-1 nodes (this will take a few minutes to complete):

gcloud container clusters create bootcamp --num-nodes 5 --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"

Learn about the deployment object

Let’s get started with Deployments. First let’s take a look at the Deployment object. The explain command in kubectl can tell us about the Deployment object.

kubectl explain deployment

We can also see all of the fields using the –recursive option.

kubectl explain deployment --recursive

You can use the explain command as you go through the lab to help you understand the structure of a Deployment object and understand what the individual fields do.

kubectl explain deployment.metadata.name

Create a deployment

Update the deployments/auth.yaml cs file:

vi deployments/auth.yaml

Start the editor:


Change the image in the containers section of the Deployment to the following:

- name: auth
  image: kelseyhightower/auth:1.0.0

Save the auth.yaml file: press <Esc> then:


Now let’s create a simple deployment. Examine the deployment configuration file:

cat deployments/auth.yaml


apiVersion: extensions/v1beta1
kind: Deployment
  name: auth
  replicas: 1
        app: auth
        track: stable
        - name: auth
          image: "kelseyhightower/auth:1.0.0"
            - name: http
              containerPort: 80
            - name: health
              containerPort: 81

Notice how the Deployment is creating one replica and it’s using version 1.0.0 of the auth container.

When you run the kubectl create command to create the auth deployment, it will make one pod that conforms to the data in the Deployment manifest. This means we can scale the number of Pods by changing the number specified in the replicas field.

Go ahead and create your deployment object using kubectl create:

kubectl create -f deployments/auth.yaml

Once you have created the Deployment, you can verify that it was created.

kubectl get deployments

Once the deployment is created, Kubernetes will create a ReplicaSet for the Deployment. We can verify that a ReplicaSet was created for our Deployment:

kubectl get replicasets

We should see a ReplicaSet with a name like auth-xxxxxxx

Finally, we can view the Pods that were created as part of our Deployment. The single Pod is created by the Kubernetes when the ReplicaSet is created.

kubectl get pods

It’s time to create a service for our auth deployment. You’ve already seen service manifest files, so we won’t go into the details here. Use the kubectl create command to create the auth service.

kubectl create -f services/auth.yaml

Now, do the same thing to create and expose the hello Deployment.

kubectl create -f deployments/hello.yaml
kubectl create -f services/hello.yaml

And one more time to create and expose the frontend Deployment.

kubectl create secret generic tls-certs --from-file tls/
kubectl create configmap nginx-frontend-conf --from-file=nginx/frontend.conf
kubectl create -f deployments/frontend.yaml
kubectl create -f services/frontend.yaml

Note: You created a ConfigMap for the frontend.

Interact with the frontend by grabbing its external IP and then curling to it.

kubectl get services frontend
curl -ks https://<EXTERNAL-IP>
curl -ks

And you get the hello response back.

You can also use the output templating feature of kubectl to use curl as a one-liner:

curl -ks https://`kubectl get svc frontend -o=jsonpath="{.status.loadBalancer.ingress[0].ip}"`

Scale a Deployment

Now that we have a Deployment created, we can scale it. Do this by updating the spec.replicas field. You can look at an explanation of this field using the kubectl explain command again.

kubectl explain deployment.spec.replicas

The replicas field can be most easily updated using the kubectl scale command:

kubectl scale deployment hello --replicas=5

Note: It may take a minute or so for all the new pods to start up.

After the Deployment is updated, Kubernetes will automatically update the associated ReplicaSet and start new Pods to make the total number of Pods equal 5.

Verify that there are now 5 hello Pods running:

kubectl get pods | grep hello- | wc -l

Now scale back the application:

kubectl scale deployment hello --replicas=3

Again, verify that you have the correct number of Pods.

kubectl get pods | grep hello- | wc -l

You learned about Kubernetes deployments and how to manage & scale a group of Pods.

Rolling update

Deployments support updating images to a new version through a rolling update mechanism. When a Deployment is updated with a new version, it creates a new ReplicaSet and slowly increases the number of replicas in the new ReplicaSet as it decreases the replicas in the old ReplicaSet.

Trigger a rolling update

To update your Deployment, run the following command:

kubectl edit deployment hello

Change the image in the containers section of the Deployment to the following:

- name: hello
  image: kelseyhightower/hello:2.0.0

Save and exit.

Once you save out of the editor, the updated Deployment will be saved to your cluster and Kubernetes will begin a rolling update.

See the new ReplicaSet that Kubernetes creates.:

kubectl get replicaset

You can also see a new entry in the rollout history:

kubectl rollout history deployment/hello

Pause a rolling update

If you detect problems with a running rollout, pause it to stop the update. Give that a try now:

kubectl rollout pause deployment/hello

Verify the current state of the rollout:

kubectl rollout status deployment/hello

You can also verify this on the Pods directly:

kubectl get pods -o jsonpath --template='{range .items[*]}{.metadata.name}{"\t"}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

Resume a rolling update

The rollout is paused which means that some pods are at the new version and some pods are at the older version. We can continue the rollout using the resume command.

kubectl rollout resume deployment/hello

When the rollout is complete, you should see the following when running the status command.

kubectl rollout status deployment/hello


deployment "hello" successfully rolled out

Rollback an update

Assume that a bug was detected in your new version. Since the new version is presumed to have problems, any users connected to the new Pods will experience those issues.

You will want to roll back to the previous version so you can investigate and then release a version that is fixed properly.

Use the rollout command to roll back to the previous version:

kubectl rollout undo deployment/hello

Verify the roll back in the history:

kubectl rollout history deployment/hello

Finally, verify that all the Pods have rolled back to their previous versions:

kubectl get pods -o jsonpath --template='{range .items[*]}{.metadata.name}{"\t"}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

Great! You learned about rolling updates for Kubernetes deployments and how to update applications without downtime.

Canary deployments

When you want to test a new deployment in production with a subset of your users, use a canary deployment. Canary deployments allow you to release a change to a small subset of your users to mitigate risk associated with new releases.

Create a canary deployment

A canary deployment consists of a separate deployment with your new version and a service that targets both your normal, stable deployment as well as your canary deployment.

First, create a new canary deployment for the new version:

cat deployments/hello-canary.yaml


apiVersion: extensions/v1beta1
kind: Deployment
  name: hello-canary
  replicas: 1
        app: hello
        track: canary
        # Use ver 1.0.0 so it matches version on service selector
        version: 1.0.0
        - name: hello
          image: kelseyhightower/hello:2.0.0
            - name: http
              containerPort: 80
            - name: health
              containerPort: 81

Make sure to update version to 1.0.0 (if your version is pointing to any other)

Now create the canary deployment:

kubectl create -f deployments/hello-canary.yaml

After the canary deployment is created, you should have two deployments, hello and hello-canary. Verify it with this kubectl command:

kubectl get deployments

On the hello service, the selector uses the app:hello selector which will match pods in both the prod deployment and canary deployment. However, because the canary deployment has a fewer number of pods, it will be visible to fewer users.

Verify the canary deployment

You can verify the hello version being served by the request:

curl -ks https://`kubectl get svc frontend -o=jsonpath="{.status.loadBalancer.ingress[0].ip}"`/version

Run this several times and you should see that some of the requests are served by hello 1.0.0 and a small subset (1/4 = 25%) are served by 2.0.0.

Canary deployments in production - session affinity

In this lab, each request sent to the Nginx service had a chance to be served by the canary deployment. But what if you wanted to ensure that a user didn’t get served by the Canary deployment? A use case could be that the UI for an application changed, and you don’t want to confuse the user. In a case like this, you want the user to “stick” to one deployment or the other.

You can do this by creating a service with session affinity. This way the same user will always be served from the same version. In the example below the service is the same as before, but a new sessionAffinity field has been added, and set to ClientIP. All clients with the same IP address will have their requests sent to the same version of the hello application.

kind: Service
apiVersion: v1
  name: "hello"
  sessionAffinity: ClientIP
    app: "hello"
    - protocol: "TCP"
      port: 80
      targetPort: 80

Due to it being difficult to set up an environment to test this, you don’t need to here, but you may want to use sessionAffinity for canary deployments in production.

Blue-green deployments

Rolling updates are ideal because they allow you to deploy an application slowly with minimal overhead, minimal performance impact, and minimal downtime. There are instances where it is beneficial to modify the load balancers to point to that new version only after it has been fully deployed. In this case, blue-green deployments are the way to go.

Kubernetes achieves this by creating two separate deployments; one for the old “blue” version and one for the new “green” version. Use your existing hello deployment for the “blue” version. The deployments will be accessed via a Service which will act as the router. Once the new “green” version is up and running, you’ll switch over to using that version by updating the Service.

A major downside of blue-green deployments is that you will need to have at least 2x the resources in your cluster necessary to host your application. Make sure you have enough resources in your cluster before deploying both versions of the application at once.

The service

Use the existing hello service, but update it so that it has a selector app:hello, version: 1.0.0. The selector will match the existing “blue” deployment. But it will not match the “green” deployment because it will use a different version.

First update the service:

kubectl apply -f services/hello-blue.yaml

Updating using Blue-Green Deployment

n order to support a blue-green deployment style, we will create a new “green” deployment for our new version. The green deployment updates the version label and the image path.

apiVersion: extensions/v1beta1
kind: Deployment
  name: hello-green
  replicas: 3
        app: hello
        track: stable
        version: 2.0.0
        - name: hello
          image: kelseyhightower/hello:2.0.0
            - name: http
              containerPort: 80
            - name: health
              containerPort: 81
              cpu: 0.2
              memory: 10Mi
              path: /healthz
              port: 81
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 15
            timeoutSeconds: 5
              path: /readiness
              port: 81
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1

Create the green deployment:

kubectl create -f deployments/hello-green.yaml

Once you have a green deployment and it has started up properly, verify that the current version of 1.0.0 is still being used:

curl -ks https://`kubectl get svc frontend -o=jsonpath="{.status.loadBalancer.ingress[0].ip}"`/version

Now, update the service to point to the new version:

kubectl apply -f services/hello-green.yaml

With the service is updated, the “green” deployment will be used immediately. You can now verify that the new version is always being used.

curl -ks https://`kubectl get svc frontend -o=jsonpath="{.status.loadBalancer.ingress[0].ip}"`/version

You did it! You learned about blue-green deployments and how to deploy updates to applications that need to switch versions all at once.

Blue-Green Rollback

If necessary, you can roll back to the old version in the same way. While the “blue” deployment is still running, just update the service back to the old version.

kubectl apply -f services/hello-blue.yaml

Once you have updated the service, your rollback will have been successful. Again, verify that the right version is now being used:

curl -ks https://`kubectl get svc frontend -o=jsonpath="{.status.loadBalancer.ingress[0].ip}"`/version

You did it! You learned about blue-green deployments and how to deploy updates to applications that need to switch versions all at once.

Build a Slack Bot with Node.js on Kubernetes

Get the sample code

In Cloud Shell on the command-line, run the following command to clone the GitHub repository:

git clone https://github.com/googlecodelabs/cloud-slack-bot.git

Change directory into cloud-slack-bot/start:

cd cloud-slack-bot/start

Install the Node.js dependencies, including Botkit:

npm install

Run the sample app

In Cloud Shell, edit the kittenbot.js file and enter your Slack bot token. If it is no longer in your clipboard, you can get it from the bot custom integration configuration page. You can use any editor you like, such as emacs or vim. This lab uses the code editor feature of Cloud Shell for simplicity.

Open the Cloud Shell code editor:

Open the Cloud Shell code editor

Open the kittenbot.js file by going to cloud-slack-bot/start/kittenbot.js:


Replace your-slack-token (Bot User OAuth Access Token) with the Slack token you copied:

var Botkit = require('botkit')

var controller = Botkit.slackbot({debug: false})
    token: 'your-slack-token' // Edit this line!
  .startRTM(function (err) {
    if (err) {
      throw new Error(err)

  ['hello', 'hi'], ['direct_message', 'direct_mention', 'mention'],
  function (bot, message) { bot.reply(message, 'Meow. :smile_cat:') })

Then save the file with Ctrl-s or File > Save.

Note: As you can see in the file, kittenbot will only respond to “hi” and “hello” greetings. Make sure you enter one of those greetings to see it respond.

In Cloud Shell run your bot:

node kittenbot.js

You may see errors in Cloud Shell, but they won’t affect the lab. It’s OK to keep going.

In your Slack workspace, you should now see that kittenbot is online. You may need to refresh your browser. Do the following to interact with it:

Click on the + next to Direct Messages:

slack direct message

Load Slack token from a file

Hard-coding the Slack token in the source code makes it likely to accidentally expose your token by publishing it to version control or embedding it in a docker image. Instead, use Kubernetes Secrets to store tokens.

You will now write your token to a file called slack-token. This filename is in the .gitignore to prevent accidentally checking it in to version control.

Create a new file in the start directory called slack-token file. (To right-click on a Chromebook: press the touchpad with two fingers, or press Alt while clicking with one finger.)

slack-token file

Copy your token from kittenbot.js or the bot configuration page, paste it into the slack-token file, then Save.


Edit the kittenbot.js file to load the Slack token specified by the slack_token_path environment variable:

var Botkit = require('botkit')
var fs = require('fs') // NEW: Add this require (for loading from files).

var controller = Botkit.slackbot({debug: false})

// START: Load Slack token from file.
if (!process.env.slack_token_path) {
  console.log('Error: Specify slack_token_path in environment')

fs.readFile(process.env.slack_token_path, function (err, data) {
  if (err) {
    console.log('Error: Specify token in slack_token_path file')
  data = String(data)
  data = data.replace(/\s/g, '')
    .spawn({token: data})
    .startRTM(function (err) {
      if (err) {
        throw new Error(err)
// END: Load Slack token from file.

  ['hello', 'hi'], ['direct_message', 'direct_mention', 'mention'],
  function (bot, message) { bot.reply(message, 'Meow. :smile_cat:') })

Go back to the Cloud Console and run your bot:

slack_token_path=./slack-token node kittenbot.js

You should see the bot online again in Slack and be able to chat with it. After testing it out, press Ctrl-c to shut down the bot.

Create a Docker container image

Docker provides a way to containerize your bot. A Docker image bundles all of your dependencies (even the compiled ones) so that it can run in a lightweight sandbox.

Building a Docker image

First, create a file called Dockerfile.

Then add the following definition, which describes how to build your Docker image.

FROM node:10.14
COPY package.json /src/package.json
RUN npm install
COPY kittenbot.js /src
CMD ["node", "/src/kittenbot.js"]

Save this file.

A Dockerfile is a recipe for a Docker image. This one layers on top of the Node.js base image found on the Docker hub, copies package.json to the image and installs the dependencies listed in it, copies the kittenbot.js file to the image, and tells Docker that it should run the Node.js server when the image starts.

Go back to the Cloud Shell and run this command to save your project ID to the environment variable PROJECT_ID. Commands in this lab will use this variable as $PROJECT_ID.

export PROJECT_ID=$(gcloud config list --format 'value(core.project)')

Build the image by running the docker build command. (This command takes about 4 minutes to complete. It has to download the base image and Node.js dependencies.)

docker build -t gcr.io/${PROJECT_ID}/slack-codelab:v1 .

While the Docker image is building, try this out this Extra Credit exercise to get a webhook to use later in the lab.

Extra Credit: Create an incoming webhook to Slack

An incoming webhook is an easy way to send Slack notifications from another service or app without having to worry about a persistent connection for two-way communication like with a bot user. Create one now:

Go to the Slack apps management page.

Click on your app Kittenbot, and then in the left panel, click Incoming Webhooks.

Toggle Activate Incoming Webhooks to On.

At the top, there’s a message “You’ve changed the permissions scopes …”. Click click here.

From the Post to dropdown, select the Slack channel #general for messages to post to.

Click Authorize.

Copy the webhook URL and save it to your computer. You’ll come back to this in a later step.

Testing a Docker image locally

Go back to Cloud Shell to test the image locally with the following command. It will run a Docker container as a daemon from the newly-created container image:

docker run -d \
    -v $(pwd)/:/config \
    -e slack_token_path=/config/slack-token \

This command also mounts the current directory as a volume inside the container to give it access to the slack-token file. You should see that kittenbot is online again.

Here’s the full documentation for the docker run command: https://docs.docker.com/engine/reference/run/

Frequently asked question: How do I get the project ID from the gcloud command-line tool?

Pushing a Docker image to Google Container Registry

Now that the image works as intended, push it to the Google Container Registry, a private repository for your Docker images accessible from every Google Cloud project (but also from outside Google Cloud Platform).

Push the Docker image to Google Container Registry from Cloud Shell:

gcloud docker -- push gcr.io/${PROJECT_ID}/slack-codelab:v1

(This command takes about 5 minutes to complete.)

Extra Credit: Testing an incoming webhook

While waiting for the image to upload, use the incoming webhook to send a notification to Slack.

In Cloud Shell, click the + button tab to add a new Cloud Shell session:

new Cloud Shell session

Go back to the incoming webhook you created. (If you closed that browser tab you can get back to there from the Slack apps management page, click your app, Kittenbot, and in the left pand, click Incoming Webhooks.)

From the Incoming Webhooks page, click the Copy button for the “Sample curl request to post to a channel”, and paste it into the new Cloud Shell Session.

This will run curl to send an HTTP request with your message to Slack.

In your Slack app, in the #general channel, look for a message from kittenbot.

This demonstrates that anywhere that you can send an HTTP request, you can send a message to Slack. This is a really easy way to integrate your own apps and services with Slack notifications.

For more complicated messages, test out the JSON request first in the Slack message builder.

Viewing images in Google Container Registry

When the image upload completes, you can see it listed in the Google Cloud Console. Under the Tools section, click on Container Registry > Images.

You now have a project-wide Docker image available which Kubernetes can access and orchestrate.

Notice a generic domain was used for the registry (gcr.io). You can also be more specific about which zone and bucket to use. Details are documented here: https://cloud.google.com/container-registry/#pushing_to_the_registry

If you’re curious, you can navigate through the container images as they are stored in Google Cloud Storage by following this link: https://console.cloud.google.com/storage/browser/.

Frequently Asked Questions:

Create a Kubernetes cluster on Kubernetes Engine

Now that the Docker image is in Google Container Registry, you can run the gcloud docker – pull command to save this image on any machine and run it with the Docker command-line tool.

If you want to make sure your bot keeps running after it is started, you’ll have to run another service to monitor your Docker container to restart it if it stops. This gets harder if you want to make sure the bot keeps running even if the machine it is running on fails.

Kubernetes solves these problems. You tell it that you want there to always be a replica of your bot running, and the Kubernetes master will keep that target state. It starts the bot up when there aren’t enough running, and shuts bot replicas down when there are too many.

A Kubernetes Engine cluster is a managed Kubernetes cluster. It consists of a Kubernetes master API server hosted by Google and a set of worker nodes. The worker nodes are Compute Engine virtual machines.

Return to the first Cloud Shell if you tried the extra credit.

Create a cluster with two n1-standard-1 nodes (this will take a few minutes to complete):

gcloud container clusters create my-cluster \
      --num-nodes=2 \
      --zone=us-central1-f \
      --machine-type n1-standard-1


Creating cluster my-cluster...done.
Created [https://container.googleapis.com/v1/projects/PROJECT_ID/zones/us-central1-f/clusters/my-cluster].
kubeconfig entry generated for my-cluster.
my-cluster  us-central1-f  n1-standard-1  2          RUNNING

Alternatively, you could create this cluster via the Cloud Console:

Kubernetes Engine > Kubernetes clusters > Create cluster.

If you use the Cloud Console to create the cluster, run gcloud container clusters get-credentials my-cluster --zone=us-central1-f to authenticate Cloud Shell with the cluster credentials.

This command creates the cluster and authenticates the Kubernetes command-line tool, kubectl, with the new cluster’s credentials.

You should now have a fully-functioning Kubernetes cluster powered by Kubernetes Engine. You’ll see it in the Cloud Console in Kubernetes Engine > Kubernetes clusters.

Each node in the cluster is a Compute Engine instance provisioned with Kubernetes and Docker binaries. If you are curious, you can list all Compute Engine instances in the project:

gcloud compute instances list


gke-my-cl...16 us-central1-f n1-standard-1 RUNNING
gke-my-cl...34 us-central1-f n1-standard-1  RUNNING

For the rest of this lab you’ll use kubectl, the Kubernetes command-line tool, to configure and run your bot.

Create a Deployment

First, create a Secret in Kubernetes to store the Slack token and make it available to the container:

kubectl create secret generic slack-token --from-file=./slack-token


secret "slack-token" created

To deploy your own containerized application to the Kubernetes cluster you need to configure a Deployment, which describes how to configure the container and provide a replication controller to keep the bot running.

In the Cloud Shell Code Editor, create a file called slack-codelab-deployment.yaml.


Enter the following deployment definition, replacing PROJECT_ID with your Project ID:

apiVersion: extensions/v1beta1
kind: Deployment
  name: slack-codelab
  replicas: 1
    type: Recreate
        app: slack-codelab
      - name: master
        image: gcr.io/PROJECT_ID/slack-codelab:v1  # Replace PROJECT_ID
                                                   # with your project ID.
        - name: slack-token
          mountPath: /etc/slack-token
        - name: slack_token_path
          value: /etc/slack-token/slack-token
      - name: slack-token
          secretName: slack-token

Save the file, File > Save.

Now you can create the Deployment by running kubectl create in Cloud Shell:

kubectl create -f slack-codelab-deployment.yaml --record


deployment "slack-codelab" created

Since you used the –record option, you can view the commands applied to this deployment as the “change-cause” in the rollout history:

kubectl rollout history deployment/slack-codelab


deployments "slack-codelab":
1               kubectl create -f slack-codelab-deployment.yaml --record

See what the kubectl create command made. Re-run this command until the status shows Running.

kubectl get pods

This should take about 30 seconds to 1 minute


NAME                             READY     STATUS    RESTARTS   AGE
slack-codelab-2890463383-1ss4a   1/1       Running   0          3m

The last things you need to do:

Frequently asked questions:

Extra Credit: Update your bot to a new version

This extra credit section should take you about 10 minutes to complete. So you’d like the bot to do more than just say “meow”. How do you deploy a new version of something that is already running on Kubernetes? First, you need to modify the application. Botkit can handle conversations. With these, the bot can request more information and react to messages beyond a one word reply.

Edit the kittenbot.js file in the code editor. Add this new kitten emoji delivery code to the bottom of the file:

// ...

// START: listen for cat emoji delivery
var maxCats = 20
var catEmojis = [

  ['cat', 'cats', 'kitten', 'kittens'],
  ['ambient', 'direct_message', 'direct_mention', 'mention'],
  function (bot, message) {
    bot.startConversation(message, function (err, convo) {
      if (err) {
      convo.ask('Does someone need a kitten delivery? Say YES or NO.', [
          pattern: bot.utterances.yes,
          callback: function (response, convo) {
            convo.ask('How many?', [
                pattern: '[0-9]+',
                callback: function (response, convo) {
                  var numCats =
                  parseInt(response.text.replace(/[^0-9]/g, ''), 10)
                  if (numCats === 0) {
                      'text': 'Sorry to hear you want zero kittens. ' +
                        'Here is a dog, instead. :dog:',
                      'attachments': [
                          'fallback': 'Chihuahua Bubbles - https://youtu.be/s84dBopsIe4',
                          'text': '<https://youtu.be/s84dBopsIe4|' +
                            'Chihuahua Bubbles>!'
                  } else if (numCats > maxCats) {
                    convo.say('Sorry, ' + numCats + ' is too many cats.')
                  } else {
                    var catMessage = ''
                    for (var i = 0; i < numCats; i++) {
                      catMessage = catMessage +
                      catEmojis[Math.floor(Math.random() * catEmojis.length)]
                default: true,
                callback: function (response, convo) {
                    "Sorry, I didn't understand that. Enter a number, please.")
          pattern: bot.utterances.no,
          callback: function (response, convo) {
            convo.say('Perhaps later.')
          default: true,
          callback: function (response, convo) {
            // Repeat the question.
  // END: listen for cat emoji delivery

Then Save the file.

You can test this out in Cloud Shell using the same command as before, but you’ll see two responses since the bot is still running in your Kubernetes cluster.

Next, build a new container image with an incremented tag (v2 in this case):

docker build -t gcr.io/${PROJECT_ID}/slack-codelab:v2 .

Push the image to Google Container Registry:

gcloud docker -- push gcr.io/${PROJECT_ID}/slack-codelab:v2

Building and pushing this updated image should be much quicker, taking full advantage of caching.

As you did with the first version of the kitten bot, you can test locally using the node command and the docker command. You can skip those steps and push the new version to the cluster.

Now you’re ready for Kubernetes to update the deployment to the new version of the application.

Edit the line in the slack-codelab-deployment.yaml file defining which image to use:

apiVersion: extensions/v1beta1
kind: Deployment
  name: slack-codelab
  replicas: 1
    type: Recreate
        app: slack-codelab
      - name: master
        image: gcr.io/PROJECT_ID/slack-codelab:v2  # Update this to v2.
                                                   # Replace PROJECT_ID
                                                   # with your project ID.
        - name: slack-token
          mountPath: /etc/slack-token
        - name: slack_token_path
          value: /etc/slack-token/slack-token
      - name: slack-token
          secretName: slack-token

Save the file.

Now, to apply this change to the running Deployment, run this command to update the deployment to use the v2 image:

kubectl apply -f slack-codelab-deployment.yaml


deployment "slack-codelab" configured

Use this code to see that Kubernetes has shut down the pod running the previous version and started a new pod that is running the new image:

kubectl get pods


NAME                             READY     STATUS        RESTARTS   AGE
slack-codelab-2890463383-mqy5l   1/1       Terminating   0          17m
slack-codelab-3059677337-b41r0   1/1       Running       0          7s

Deployment Strategies: Since the deployment strategy was specified to “recreate” when you created the deployment, Kubernetes makes sure the old instances are shut down before creating a new one. This strategy is used because:

See the changed deployment:

kubectl rollout history deployment/slack-codelab


deployments "slack-codelab":
1               kubectl create -f slack-codelab-deployment.yaml --record
2               kubectl apply -f slack-codelab-deployment.yaml

Go back to Slack and type a message to kittenbot that mentions “kitten” and see it join the conversation.

You just updated a Slack bot running on Kubernetes to a new version.

Using Kubernetes Engine to Deploy Apps with Regional Persistent Disks


In this lab you will learn how to configure a highly available application by deploying WordPress using regional persistent disks on Kubernetes Engine. Regional persistent disks provide synchronous replication between two zones, which keeps your application up and running in case there is an outage or failure in a single zone. Deploying a Kubernetes Engine Cluster with regional persistent disks will make your application more stable, secure, and reliable.

Using Kubernetes Engine to Deploy Apps with Regional Persistent Disks - overview

What you’ll do


This is an advanced lab. Before taking it, you should be familiar with at least the basics of Kubernetes and WordPress. Here are some Qwiklabs that can get you up to speed:

Once you’re prepared, scroll down to get your lab environment set up.

Creating the Regional Kubernetes Engine Cluster

Open a new Cloud Shell session. You will first create a regional Kubernetes Engine cluster that spans three zones in the us-west1 region. First, fetch the server configuration for the us-west1 region and export environment variables by running:

CLUSTER_VERSION=$(gcloud container get-server-config --region us-west1 --format='value(validMasterVersions[0])')


Now create a standard Kubernetes Engine cluster (this will take a little while, ignore any warnings about node auto repairs):

gcloud container clusters create repd \
  --cluster-version=${CLUSTER_VERSION} \
  --machine-type=n1-standard-4 \
  --region=us-west1 \
  --num-nodes=1 \

Example Output:

Creating cluster repd...done.
Created [https://container.googleapis.com/v1beta1/projects/qwiklabs-gcp-e8f5f22705c770ab/zones/us-west1/clusters/repd].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-west1/repd?project=qwiklabs-gcp-e8f5f22705c770ab
kubeconfig entry generated for repd.
repd  us-west1  1.12.6-gke.7  n1-standard-4  1.12.6-gke.7  3          RUNNING

You just created a regional cluster (located in us-west1) with one node in each zone (us-west1-a,us-west1-b,us-west1-c). Navigate to Compute Engine from the left-hand menu to view your instances:

Navigate to Compute Engine from the left-hand menu to view your instances

Deploying the App with a Regional Disk

Now that you have your Kubernetes cluster running, you’ll do the following three things:

Install and initialize Helm to install the chart package

The chart package, which is installed with Helm, contains everything you need to run WordPress.

  1. Install Helm locally in your Cloud Shell instance by running:
    curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh
    chmod 700 get_helm.sh

    Initialize Helm: ``` kubectl create serviceaccount tiller –namespace kube-system

kubectl create clusterrolebinding tiller-cluster-rule \ –clusterrole=cluster-admin \ –serviceaccount=kube-system:tiller helm init –service-account=tiller until (helm version –tiller-connection-timeout=1 >/dev/null 2>&1); do echo “Waiting for tiller install…”; sleep 2; done && echo “Helm install complete”

Helm is now installed in your cluster.

#### Create the StorageClass
Next you'll create the StorageClass used by the chart to define the zones of the regional disk. The zones listed in the StorageClass will match the zones of the Kubernetes Engine cluster.

Create a StorageClass for the regional disk by running:

kubectl apply -f - «EOF kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: repd-west1-a-b-c provisioner: kubernetes.io/gce-pd parameters: type: pd-standard replication-type: regional-pd zones: us-west1-a, us-west1-b, us-west1-c EOF

Example Output:

storageclass “repd-west1-a-b-c” created

You now have a StorageClass that is capable of provisioning PersistentVolumes that are replicated across the us-west1-a, us-west1-b and us-west1-c zones.

List the available `storageclass` with:

kubectl get storageclass

Example Output:

NAME PROVISIONER AGE repd-west1-a-b-c kubernetes.io/gce-pd 26s standard (default) kubernetes.io/gce-pd 1h

### Create Persistent Volume Claims
In this section, you will create persistentvolumeclaims for your application.

Create data-wp-repd-mariadb-0 PVC with standard StorageClass.

kubectl apply -f - «EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: data-wp-repd-mariadb-0 namespace: default labels: app: mariadb component: master release: wp-repd spec: accessModes: - ReadOnlyMany resources: requests: storage: 8Gi storageClassName: standard EOF

Create `wp-repd-wordpress` PVC with `repd-west1-a-b-c` StorageClass.

kubectl apply -f - «EOF kind: PersistentVolumeClaim apiVersion: v1 metadata: name: wp-repd-wordpress namespace: default labels: app: wp-repd-wordpress chart: wordpress-5.7.1 heritage: Tiller release: wp-repd spec: accessModes: - ReadOnlyMany resources: requests: storage: 200Gi storageClassName: repd-west1-a-b-c EOF

List the available `persistentvolumeclaims` with:

kubectl get persistentvolumeclaims

Example Output:

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-wp-repd-mariadb-0 Bound pvc-8a10ed04-56ca-11e9-a020-42010a8a003d 8Gi ROX standard 21m wp-repd-wordpress Bound pvc-ad5ddb0b-56ca-11e9-9af5-42010a8a0047 200Gi ROX repd-west1-a-b-c 20m

Deploy WordPress
Now that we have our StorageClass configured, Kubernetes automatically attaches the persistent disk to an appropriate node in one of the available zones.

1. Deploy the WordPress chart that is configured to use the StorageClass that you created earlier:

helm install –name wp-repd \ –set smtpHost= –set smtpPort= –set smtpUser= \ –set smtpPassword= –set smtpUsername= –set smtpProtocol= \ –set persistence.storageClass=repd-west1-a-b-c \ –set persistence.existingClaim=wp-repd-wordpress \ –set persistence.accessMode=ReadOnlyMany \ stable/wordpress


NAME: wp-repd LAST DEPLOYED: Fri May 24 09:14:55 2019 NAMESPACE: default STATUS: DEPLOYED

RESOURCES: ==> v1/ConfigMap NAME DATA AGE wp-repd-mariadb 1 1s wp-repd-mariadb-tests 1 1s

==> v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE wp-repd-wordpress 0/1 1 0 1s

==> v1/Pod(related) NAME READY STATUS RESTARTS AGE wp-repd-mariadb-0 0/1 ContainerCreating 0 1s wp-repd-wordpress-7889b789f-l2skn 0/1 ContainerCreating 0 1s

==> v1/Secret NAME TYPE DATA AGE wp-repd-mariadb Opaque 2 1s wp-repd-wordpress Opaque 1 1s

==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE wp-repd-mariadb ClusterIP 3306/TCP 1s wp-repd-wordpress LoadBalancer 80:32453/TCP,443:30153/TCP 1s

==> v1beta1/StatefulSet NAME READY AGE wp-repd-mariadb 0/1 1s


  1. Get the WordPress URL:

NOTE: It may take a few minutes for the LoadBalancer IP to be available. Watch the status with: ‘kubectl get svc –namespace default -w wp-repd-wordpress’ export SERVICE_IP=$(kubectl get svc –namespace default wp-repd-wordpress –template “”) echo “WordPress URL: http://$SERVICE_IP/” echo “WordPress Admin URL: http://$SERVICE_IP/admin”

  1. Login with the following credentials to see your blog

echo Username: user echo Password: $(kubectl get secret –namespace default wp-repd-wordpress -o jsonpath=”{.data.wordpress-password}” | base64 –decode)

2. List out available wordpress pods:

kubectl get pods

Example Output:

NAME READY STATUS RESTARTS AGE wp-repd-mariadb-79444cd49b-lx8jq 1/1 Running 0 35m wp-repd-wordpress-7654c85b66-gz6nd 1/1 Running 0 35m

3. Run the following command which waits for the service load balancer's external IP address to be created:

while [[ -z $SERVICE_IP ]]; do SERVICE_IP=$(kubectl get svc wp-repd-wordpress -o jsonpath=’{.status.loadBalancer.ingress[].ip}’); echo “Waiting for service external IP…”; sleep 2; done; echo http://$SERVICE_IP/admin

4. Verify that the persistent disk was created:

while [[ -z $PV ]]; do PV=$(kubectl get pvc wp-repd-wordpress -o jsonpath=’{.spec.volumeName}’); echo “Waiting for PV…”; sleep 2; done

kubectl describe pv $PV

### Simulating a zone failure
Next you will simulate a zone failure and watch Kubernetes move your workload to the other zone and attach the regional disk to the new node.

1. Obtain the current node of the WordPress pod:

NODE=$(kubectl get pods -l app=wp-repd-wordpress -o jsonpath=’{.items..spec.nodeName}’)

ZONE=$(kubectl get node $NODE -o jsonpath=”{.metadata.labels[‘failure-domain.beta.kubernetes.io/zone’]}”)

IG=$(gcloud compute instance-groups list –filter=”name~gke-repd-default-pool zone:(${ZONE})” –format=’value(name)’)

echo “Pod is currently on node ${NODE}”

echo “Instance group to delete: ${IG} for zone: ${ZONE}”

Example Output:

Pod is currently on node gke-repd-default-pool-b8cf37cd-bc5q Instance group to delete: gke-repd-default-pool-b8cf37cd-grp for zone: us-west1-c

You can also verify it with:

kubectl get pods -l app=wp-repd-wordpress -o wide

Example Output:

NAME READY STATUS RESTARTS AGE IP NODE wp-repd-wordpress-7654c85b66-gz6nd 1/1 Running 0 1h gke-repd-default-pool-b8cf37cd-bc5q

Take note of `Node` column. You are going to delete this node to simulate the zone failure.

2. Now run the following to delete the instance group for the node where the WordPress pod is running, click **Y** to continue deleting:

gcloud compute instance-groups managed delete ${IG} –zone ${ZONE}

Kubernetes is now detecting the failure and migrates the pod to a node in another zone.

3. Verify that both the WordPress pod and the persistent volume migrated to the node that is in the other zone:

kubectl get pods -l app=wp-repd-wordpress -o wide

Example Output:

NAME READY STATUS RESTARTS AGE IP NODE wp-repd-wordpress-7654c85b66-xqb78 1/1 Running 0 1m gke-repd-default-pool-9da1b683-h70h

Make sure the node that is displayed is different from the node in the previous step.

4. Once the new service has a Running status, open the WordPress admin page in your browser from the link displayed in the command output:

echo http://$SERVICE_IP/admin

You have attached a regional persistent disk to a node that is in a different zone.

## NGINX Ingress Controller on Google Kubernetes Engine

### Set a zone
Before creating a Kubernetes cluster, we'll have to set a default computing zone for our GCP project. Run the following command to see a [list of GCP zones](https://cloud.google.com/compute/docs/regions-zones/):

gcloud compute zones list

Now run the following command to set your zone (in this case to us-central1-a):

gcloud config set compute/zone us-central1-a

### Create a Kubernetes cluster
Now that our zone is configured, let's deploy a Kubernetes Engine cluster. Run the following command to create a cluster named nginx-tutorial that's made up of two nodes (or worker machines):

gcloud container clusters create nginx-tutorial –num-nodes 2

It will take a few minutes for this command to complete. Continue when you get a similar output in Cloud Shell:

![Oreate a Kubernetes cluster](/assets/images/kubernetes-02/sGVbE9DJ7cqbEpEZPJhAZ94i+66rLtiNheu22Kru0sk.png)

### Install Helm
Now that we have our Kubernetes cluster up and running, let's install [Helm](https://helm.sh/). Helm is a tool that streamlines Kubernetes application installation and management. You can think of it like apt, yum, or homebrew for Kubernetes. Using helm charts is recommended, since they are maintained and typically kept up-to-date by the Kubernetes community. Helm has two parts: a client (helm) and a server (tiller):

* **Tiller** runs inside your Kubernetes cluster and manages releases (installations) of your Helm Charts.
* **Helm** runs on your laptop, CI/CD, or in this case, the Cloud Shell.

Helm comes preconfigured with an installer script that automatically grabs the latest version of the Helm client and installs it locally. Fetch the script by running the following command:

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

Next, run the following commands to get the Helm client installed:

chmod 700 get_helm.sh ./get_helm.sh

Now initialize helm:

helm init

Great! You now have the latest copy of the Helm client installed and ready for use in your Cloud Shell environment.

### Installing Tiller
Starting with Kubernetes v1.8+, [RBAC](https://en.wikipedia.org/wiki/Role-based_access_control) is enabled by default. Prior to installing tiller you need to ensure that you have the correct ServiceAccount and ClusterRoleBinding configured for the tiller service. This allows tiller to be able to install services in the default namespace.

Run the following commands to install the server-side tiller to the Kubernetes cluster with RBAC enabled:

kubectl create serviceaccount –namespace kube-system tiller

kubectl create clusterrolebinding tiller-cluster-rule –clusterrole=cluster-admin –serviceaccount=kube-system:tiller

kubectl patch deploy –namespace kube-system tiller-deploy -p ‘{“spec”:{“template”:{“spec”:{“serviceAccount”:”tiller”}}}}’

Now initialize Helm with your newly-created service account:

helm init –service-account tiller –upgrade

You can also confirm that tiller is running by checking for the tiller-deploy Deployment in the kube-system namespace. Run the following command to do so:

kubectl get deployments -n kube-system

The output should have a tiller-deploy Deployment as shown below:

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE event-exporter-v0.1.7 1 1 1 1 13m heapster-v1.4.3 1 1 1 1 13m kube-dns 2 2 2 2 13m kube-dns-autoscaler 1 1 1 1 13m kubernetes-dashboard 1 1 1 1 13m l7-default-backend 1 1 1 1 13m tiller-deploy 1 1 1 1 4m

### Deploy an application in Kubernetes Engine
Now that you have Helm configured, let's deploy a simple web-based application from the Google Cloud Repository. This application will be used as the backend for the Ingress.

From the Cloud Shell, run the following command:

kubectl run hello-app –image=gcr.io/google-samples/hello-app:1.0 –port=8080

Your output should resemble the following:

deployment “hello-app” created

Now expose the hello-app Deployment as a Service by running the following command:

kubectl expose deployment hello-app

Your output should resemble the following:

service “hello-app” exposed

### Deploying the NGINX Ingress Controller via Helm

The Kubernetes platform gives administrators flexibility when it comes to Ingress Controllers—you can integrate your own rather than having to work with your provider's built-in offering. The NGINX controller must be exposed for external access. This is done using Service `type: LoadBalancer` on the NGINX controller service. On Kubernetes Engine, this creates a Google Cloud Network (TCP/IP) Load Balancer with NGINX controller Service as a backend. Google Cloud also creates the appropriate firewall rules within the Service's VPC to allow web HTTP(S) traffic to the load balancer frontend IP address.

#### NGINX Ingress Controller on Kubernetes Engine
The following flowchart is a visual representation of how an NGINX controller runs on a Kubernetes Engine cluster:

![NGINX Ingress Controller on Kubernetes Engine](/assets/images/kubernetes-02/E1bWzR59KZ8Yjlwzj9Jm5M138L1Jwn_ydY9cZdD2A4U.png)

#### Deploy NGINX Ingress Controller

Now that you have the bigger picture in mind, let's go ahead and deploy the NGINX Ingress Controller. Run the following command to do so:

helm install –name nginx-ingress stable/nginx-ingress –set rbac.create=true

In the output under RESOURCES, you should see a similar output:

==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-ingress-controller LoadBalancer 80:31110/TCP,443:31771/TCP 1s nginx-ingress-default-backend ClusterIP 80/TCP 1s

Note the second service, `nginx-ingress-default-backend`. The default backend is a Service which handles all URL paths and hosts the NGINX controller. The default backend exposes two URLs:

* `/healthz` that returns 200
* `/` that returns 404

Wait a few moments while the GCP L4 Load Balancer gets deployed. Confirm that the `nginx-ingress-controller` Service has been deployed and that you have an external IP address associated with the service by running the following command:

kubectl get service nginx-ingress-controller

You receive a similar output:

nginx-ingress-controller LoadBalancer

### Configure Ingress Resource to use NGINX Ingress Controller
An Ingress Resource object is a collection of L7 rules for routing inbound traffic to Kubernetes Services. Multiple rules can be defined in one Ingress Resource or they can be split up into multiple Ingress Resource manifests. The Ingress Resource also determines which controller to utilize to serve traffic. This can be set with an annotation, `kubernetes.io/ingress.class`, in the metadata section of the Ingress Resource. For the NGINX controller, you will use the nginx value as shown below:

annotations: kubernetes.io/ingress.class: nginx

On Kubernetes Engine, if no annotation is defined under the metadata section, the Ingress Resource uses the GCP GCLB L7 load balancer to serve traffic. This method can also be forced by setting the annotation's value to gce, like below:

annotations: kubernetes.io/ingress.class: gce

Let's create a simple Ingress Resource YAML file which uses the NGINX Ingress Controller and has one path rule defined by typing the following commands:

touch ingress-resource.yaml nano ingress-resource.yaml

Add the following content in `ingress-resource.yaml` file:

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-resource annotations: kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/ssl-redirect: “false” spec: rules:

then press Ctrl-X, then press Y, then press Enter to save the file.

The kind: Ingress dictates it is an Ingress Resource object. This Ingress Resource defines an inbound L7 rule for path /hello to service hello-app on port 8080.

Run the following command to apply those rules to our Kubernetes application:

kubectl apply -f ingress-resource.yaml

Verify that Ingress Resource has been created:

kubectl get ingress ingress-resource

Note: The IP address for the Ingress Resource will not be defined right away. Wait a few moments for the ADDRESS field to get populated.

Test Ingress and default backend

You should now be able to access the web application by going to the EXTERNAL-IP/hello address of the NGINX ingress controller (found by running kubectl get service nginx-ingress-controller).

Open a new tab and go to the following, replacing the external-ip-of-ingress-controller with the external IP address of the NGINX ingress controller:


Your page should look similar to the following:

Test Ingress and default backend

To check if the default-backend service is working properly, access any path (other than the path /hello defined in the Ingress Resource) and ensure you receive a 404 message. For example:


Your page should look similar to the following:

To check if the default-backend service is working properly

Distributed Load Testing Using Kubernetes


In this lab you will learn how to use Kubernetes Engine to deploy a distributed load testing framework. The framework uses multiple containers to create load testing traffic for a simple REST-based API. Although this solution tests a simple web application, the same pattern can be used to create more complex load testing scenarios such as gaming or Internet-of-Things (IoT) applications. This solution discusses the general architecture of a container-based load testing framework.

System under test

For this lab the system under test is a small web application deployed to Google App Engine. The application exposes basic REST-style endpoints to capture incoming HTTP POST requests (incoming data is not persisted).

Example workloads

The application that you’ll deploy is modeled after the backend service component found in many Internet-of-Things (IoT) deployments. Devices first register with the service and then begin reporting metrics or sensor readings, while also periodically re-registering with the service.

Common backend service component interaction looks like this:

Common backend service component interaction

To model this interaction, you’ll use Locust, a distributed, Python-based load testing tool that is capable of distributing requests across multiple target paths. For example, Locust can distribute requests to the /login and /metrics target paths.

The workload is based on the interaction described above and is modeled as a set of Tasks in Locust. To approximate real-world clients, each Locust task is weighted. For example, registration happens once per thousand total client requests.

Container-based computing

A particular pod can disappear for a variety of reasons, including node failure or intentional node disruption for updates or maintenance. This means that the IP address of a pod does not provide a reliable interface for that pod. A more reliable approach would use an abstract representation of that interface that never changes, even if the underlying pod disappears and is replaced by a new pod with a different IP address. A Kubernetes Engine service provides this type of abstract interface by defining a logical set of pods and a policy for accessing them.

In this lab there are several services that represent pods or sets of pods. For example, there is a service for the DNS server pod, another service for the Locust master pod, and a service that represents all 10 Locust worker pods.

The following diagram shows the contents of the master and worker nodes:

the contents of the master and worker nodes

What you’ll do


Set project and zone

Define environment variables for the project id, region and zone you want to use for the lab.

PROJECT=$(gcloud config get-value project)
gcloud config set compute/region $REGION
gcloud config set compute/zone $ZONE

Get the sample code and build a Docker image for the application

Get the source code from the repository by running:

git clone https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubernetes.git

Move into the directory:

cd distributed-load-testing-using-kubernetes/

Build docker image and store it in container registry.

gcloud builds submit --tag gcr.io/$PROJECT/locust-tasks:latest docker-image/.

Example Output:

ID                                    CREATE_TIME                DURATION  SOURCE                                                                                                   IMAGES                                                       STATUS
47f1b8f7-0b81-492c-aa3f-19b2b32e515d  xxxxxxx  51S       gs://project_id_cloudbuild/source/1554261539.12-a7945015d56748e796c55f17b448e368.tgz  gcr.io/project_id/locust-tasks (+1 more)  SUCCESS

Deploy Web Application

The sample-webapp folder contains a simple Google App Engine Python application as the “system under test”. To deploy the application to your project use the gcloud app deploy command:

gcloud app deploy sample-webapp/app.yaml

Note: You will need the URL of the deployed sample web application when deploying the locust-master and locust-worker deployments which is already stored in TARGET variable.

Deploy Kubernetes Cluster

First create the Google Kubernetes Engine cluster using the gcloud command shown below:

gcloud container clusters create $CLUSTER \
  --zone $ZONE \

Example Output:

gke-load-test  us-central1-a  1.11.7-gke.12  n1-standard-1  1.11.7-gke.12  5          RUNNING

Load testing master

The first component of the deployment is the Locust master, which is the entry point for executing the load testing tasks described above. The Locust master is deployed with a single replica because we need only one master.

The configuration for the master deployment specifies several elements, including the ports that need to be exposed by the container (8089 for web interface, 5557 and 5558 for communicating with workers). This information is later used to configure the Locust workers.

The following snippet contains the configuration for the ports:

   - name: loc-master-web
     containerPort: 8089
     protocol: TCP
   - name: loc-master-p1
     containerPort: 5557
     protocol: TCP
   - name: loc-master-p2
     containerPort: 5558
     protocol: TCP

Deploy locust-master

Replace [TARGET_HOST] and [PROJECT_ID] in locust-master-controller.yaml and locust-worker-controller.yaml with the deployed endpoint and project-id respectively.

sed -i -e "s/\[TARGET_HOST\]/$TARGET/g" kubernetes-config/locust-master-controller.yaml
sed -i -e "s/\[TARGET_HOST\]/$TARGET/g" kubernetes-config/locust-worker-controller.yaml
sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kubernetes-config/locust-master-controller.yaml
sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kubernetes-config/locust-worker-controller.yaml

Deploy Locust master:

kubectl apply -f kubernetes-config/locust-master-controller.yaml

To confirm that the locust-master pod is created, run the following command:

kubectl get pods -l app=locust-master

Next, deploy the locust-master-service:

kubectl apply -f kubernetes-config/locust-master-service.yaml

This step will expose the pod with an internal DNS name (locust-master) and ports 8089, 5557, and 5558. As part of this step, the type: LoadBalancer directive in locust-master-service.yaml will tell Google Kubernetes Engine to create a Google Compute Engine forwarding-rule from a publicly avaialble IP address to the locust-master pod.

To view the newly created forwarding-rule, execute the following:

kubectl get svc locust-master

Example Output:

NAME            TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)                                        AGE
locust-master   LoadBalancer   8089:30865/TCP,5557:30707/TCP,5558:31327/TCP   1m

Load testing workers

The next component of the deployment includes the Locust workers, which execute the load testing tasks described above. The Locust workers are deployed by a single deployment that creates multiple pods. The pods are spread out across the Kubernetes cluster. Each pod uses environment variables to control important configuration information such as the hostname of the system under test and the hostname of the Locust master.

After the Locust workers are deployed, you can return to the Locust master web interface and see that the number of slaves corresponds to the number of deployed workers.

The following snippet contains the deployment configuration for the name, labels, and number of replicas:

apiVersion: "extensions/v1beta1"
kind: "Deployment"
  name: locust-worker
    name: locust-worker
  replicas: 5
      app: locust-worker
        app: locust-worker

Deploy locust-worker

Now deploy locust-worker-controller:

kubectl apply -f kubernetes-config/locust-worker-controller.yaml

The locust-worker-controller is set to deploy 5 locust-worker pods. To confirm they were deployed run the following:

kubectl get pods -l app=locust-worker

Scaling up the number of simulated users will require an increase in the number of Locust worker pods. To increase the number of pods deployed by the deployment, Kubernetes offers the ability to resize deployments without redeploying them.

The following command scales the pool of Locust worker pods to 20:

kubectl scale deployment/locust-worker --replicas=20

o confirm that pods have launched and are ready, get the list of locust-worker pods:

kubectl get pods -l app=locust-worker

The following diagram shows the relationship between the Locust master and the Locust workers:

the relationship between the Locust master and the Locust workers

Execute Tests

To execute the Locust tests, get the external IP address by following command:

EXTERNAL_IP=$(kubectl get svc locust-master -o yaml | grep ip | awk -F": " '{print $NF}')
echo http://$EXTERNAL_IP:8089

Click the link and navigate to Locust master web interface.

Running Dedicated Game Servers in Google Kubernetes Engine

Packaging server applications as containers is quickly gaining traction across the tech landscape, and game companies are no exception. Many game companies are interested in using containers to improve VM utilization as well as take advantage of their isolated run-time paradigm. Despite high interest, many companies don’t know where to start.

This lab will show you how to use an expandable architecture for running a real-time, session-based multiplayer dedicated game server using Kubernetes on Google Kubernetes Engine. A scaling manager process is configured to automatically start and stop virtual machine instances as needed. Configuration of the machines as Kubernetes nodes is handled automatically by managed instance groups. The online game structure presented in this lab is intentionally simple, and places where additional complexity might be useful or necessary are pointed out where appropriate.


Before you begin start the download of the game client

In this lab you’ll be creating a game server, and to test the server, you need to connect to a game client. The OpenArena game client is available for many operating systems, as long as you can install it on your local machine. If you cannot install a game client on the machine you’re currently using, consider taking this lab when you are using a machine you can install a game client on. Validating your work by connecting to a game client is not required, you can choose not to. Taking this lab will still show you how to create a dedicated game server, you just won’t be able to confirm it works.

Install the OpenArena game client on your computer to test the connection to the game server at the end of the lab :


This will take a while. Start working on the lab while it downloads, and check back in about 30 min to see if it’s finished. Then install the game client.

Architecture overview

The Overview of Cloud Game Infrastructure discusses the high-level components common to many online game architectures. In this lab you implement a Kubernetes DGS cluster frontend service and a scaling manager backend service. A full production game infrastructure would also include many other frontend and backend services which are outside the scope of this solution.

Architecture overview

Constructive constraints

In an effort to produce an example that is both instructive and simple enough to extend, this lab assumes the following game constraints:

Note: Many of the commands in this lab require you to substitute your own values for those listed in the code. These substituted variables are capitalized and surrounded by angle brackets <>, such as .

Containerizing the dedicated game server

In this lab, you’ll use OpenArena, a “community-produced deathmatch FPS based on GPL idTech3 technology”. Although this game’s technology is over fifteen years old, it’s still a good example of a common DGS pattern:

This architecture has many benefits: it speeds image distribution, reduces the update load as only binaries are replaced, and consumes less disk space.

Creating a dedicated game server binaries container image

For this lab, you cannot use Cloud Shell. You must install the GCP SDK in a client and prepare the environment so that you can create and work with docker images.

Create a VM instance to carry out lab tasks

In the Google Cloud Console Click Compute Engine > VM Instances.

Click Create Instance.

In the Identity and API access section select Allow full access to all Cloud APIs.

Click Create.

After your instance has deployed, click the SSH button. The remaining tasks for this lab will be carried out in the SSH console of this VM.

If the SSH console does not automatically open check the upper right of your browser window for a pop-up alert icon.

Click the pop-up alert icon, select Always allow pop-ups from https://console.cloud.google.com and then click Done.

Click the SSH button again to open the console.

Install kubectl and docker on your VM

Update the apt-get repositories in the VM:

sudo apt-get update

Install gcloud tools for Kubernetes:

sudo apt-get -y install kubectl

Install dependencies for docker:

sudo apt-get -y install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \

Install Docker’s official GPG key:

curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -

Check that the GPG fingerprint of the Docker key you downloaded matches the expected value:

sudo apt-key fingerprint 0EBFCD88

You should see the following fingerprint in the output

pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

Add the stable Docker repository:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")    $(lsb_release -cs) stable"

Update the apt-get repositories:

sudo apt-get update

Install Docker-ce:

sudo apt-get -y install docker-ce

Configure Docker to run as a non-root user:

sudo usermod -aG docker $USER

Exit from the SSH session then reconnect by clicking the SSH button for your VM instance.

You have to disconnect and reconnect the SSH session because the group permission change does not take effect until the user logs in again.

Confirm that you can run Docker as a non-root user:

docker run hello-world

You should see the following message in the output.

Hello from Docker!
This message shows that your installation appears to be working correctly.

Download the Sample Game Server Demo

Clone the demo example:

git clone https://github.com/GoogleCloudPlatform/gke-dedicated-game-server.git

Creating a dedicated game server binaries container image

Prepare the working environment Select the gcr.io region nearest to your Kubernetes Engine cluster (for example, us for the United States, eu for Europe, or asia for Asia, as noted in the documentation).

Substitute your <PROJECT_ID> and your gcr.io region for <GCR_REGION> in this command:

printf "$GCR_REGION \n$PROJECT_ID\n"

The updated command will look something like:

export GCR_REGION=us PROJECT_ID=qwiklabs-gcp-6bcf12ce01eb3dc7
printf "$GCR_REGION \n$PROJECT_ID\n"

Generate a new container image

You first create a Dockerfile describing the image to be built. This lab’s Debian-based Dockerfile is in the repository at openarena/Dockerfile.

cd gke-dedicated-game-server/openarena

Run this Docker build command to generate the container image and tag it. For ease of use with Google Kubernetes Engine, we’re using Google Container Registry ( gcr.io ).

docker build -t \
${GCR_REGION}.gcr.io/${PROJECT_ID}/openarena:0.8.8 .

You may see some warnings and errors during the build but if the command completes with two statements starting with Successfully built and Successfully tagged then you can proceed.

Upload the container image to an image repository:

gcloud docker -- push \

Generate an assets disk

In most games binaries are orders of magnitude smaller than assets. Because of this fact, it makes sense to create a container image that only contains binaries; assets can be put on a persistent disk and attached to multiple VM instances that run the DGS container. This architecture saves money and eliminates the need to distribute assets to all VM instances.

Create an OpenArena asset disk by following these steps:

Set the default region and store a zone ID in an environment variable:

gcloud config set compute/region ${region}

Create a small Compute Engine VM instance using gcloud.

gcloud compute instances create openarena-asset-builder \
   --machine-type f1-micro \
   --image-family debian-9 \
   --image-project debian-cloud \
   --zone ${zone_1}

Create and attach an appropriately-sized persistent disk. The persistent disk must be separate from the boot disk, and should be configured to remain undeleted when the virtual machine is removed. Kubernetes persistentVolume functionality works best with persistent disks initialized according to the Compute Engine documentation consisting of a single ext4 file system without a partition table.

gcloud compute disks create openarena-assets \
   --size=50GB --type=pd-ssd\
   --description="OpenArena data disk. Mount read-only at
/usr/share/games/openarena/baseoa/" \
   --zone ${zone_1}

Note: Some persistent disk types have more performance when provisioned at larger disk sizes. Please consult the documentation when choosing your disk size. Depending on how read-heavy your engine is, it might make sense to choose a disk larger than needed to hold the data in order to have additional performance.

Wait until the openarena-asset-builder instance has fully started up, then attach the persistent disk.

gcloud compute instances attach-disk openarena-asset-builder \
   --disk openarena-assets --zone ${zone_1}

Once attached, you can SSH into the openarena-asset-builder VM instance and format the new persistent disk.

Connect to the Asset Builder VM Instance using SSH

In the Console, click Compute Engine > VM Instances.

Click the SSH button next to the openarena-asset-builder instance.

This will open a new SSH console that you will use to prepare the persistent volume.

Format and Configure the Assets Disk

New disks are unformatted. You must format and mount a disk before it can be used. First, attach the openarena-assets persistent disk to the openarena-asset-builder VM instance.

Because the mkfs.ext4 command in the next step is a destructive command, confirm the device ID for the openarena-assets disk. If you’re following this lab from the beginning, the ID is /dev/sdb.

Verify this using the lsblk command to look at the attached disks and their partitions:

sudo lsblk

The output should show the 10 GB OS disk sda with 1 partition sda1 and the 50 GB openarena-assets disk with no partition as device sdb:

sda      8:0  0  10G  0 disk
└─sda1   8:1  0  10G  0 part /
sdb      8:16 0  50G  0 disk

Note: If you only see sda and sda1 in this list, check that you are running these commands in the SSH console for the openarena-assets-builder VM instance.

Format the openarena-assets disk:

sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

Install OpenArena on the openarena-asset-builder VM instance and copy the compressed asset archives to the openarena-assets persistent disk. For this game, the assets are .pk3 files, located in the /usr/share/games/openarena/baseoa/ directory.

To save some work, mount the assets disk to this directory before installing, so all the .pk3 files get put on the disk by the install process:

sudo mkdir -p /usr/share/games/openarena/baseoa/

sudo mount -o discard,defaults /dev/sdb \
sudo apt-get update
sudo apt-get -y install openarena-server

sudo gsutil cp gs://qwiklabs-assets/single-match.cfg /usr/share/games/openarena/baseoa/single-match.cfg

Once the installation has been completed unmount the persistent volume and shut down the instance:

sudo umount -f -l /usr/share/games/openarena/baseoa/
sudo shutdown -h now

The SSH console will now stop responding, this is expected.

The persistent disk is ready to be used as a persistentVolume in Kubernetes and the instance can be safely deleted.

Return to your main lab VM instance SSH console and verify value of zone_1 variable which you set earlier

echo $zone_1

Expected output:


If it is not set, i.e. it returns none, you can set it via following command:


Now, delete the openarena-asset-builder VM.

gcloud compute instances delete openarena-asset-builder --zone ${zone_1}

Enter Y when prompted to confirm the deletion.

When implementing persistent disks as part of a game development pipeline, configure your build system to create the persistent disk with all the asset files in an appropriate directory structure. This may take the form of a simple script running gcloud commands, or a GCP-specific plugin for your build system of choice. It’s also recommended that you create multiple copies of the persistent disk and have VM instances connected to these copies in a balanced manner both for throughput considerations and to manage failure risk.

Setting up a Kubernetes cluster

Creating a Kubernetes cluster on Container Engine For this lab you’re using a standard Container Engine cluster with the n1-standard machine type, which fine for demonstration purposes. In a production environment, the n1-highcpu machine types are more appropriate for the usage profile of OpenArena. The number of vCPUs you’ll run on each machine is largely influenced by two factors:

For this lab the number of vCPUs is limited to 2 per node as that is sufficient.

In Cloud Shell create a network and firewall rules for the gaming cluster.

gcloud compute networks create game
gcloud compute firewall-rules create openarena-dgs --network game \
    --allow udp:27961-28061

Use SSH console for the main VM instance which you created earlier to run next commands.

The following gcloud create command creates a three-node cluster ( –numnodes 3 ) with four virtual CPU cores on each ( –machine-type=n1-standard-2 ):

gcloud container clusters create openarena-cluster \
   --num-nodes 3 \
   --network game \
   --machine-type=n1-standard-2 \

It may take up to 10 minutes for the cluster to start up.

Once the cluster has started, set up your local shell with the proper Kubernetes authentication credentials to control your new cluster

gcloud container clusters get-credentials openarena-cluster --zone ${zone_1}

Although the managed instance groups feature used by Container Engine offers VM instance autoscaling to increase and decrease the number of nodes in the pool based on usage, the previous command disabled this feature using the --disable-addons=HttpLoadBalancing,HorizontalPodAutoscaling flag.

Many game developers find it useful to implement a custom scaling manager process which is DGS aware to deal with the specific requirements of this type of workload. The managed instance group does serve an important function: its default Container Engine image template includes all the necessary Kubernetes software and automatically registers the node with the master on startup.

In addition, you disabled HTTP Load Balancing with the –disable-addons flag. The pods running in this cluster don’t require it.

Configuring the assets disk in Kubernetes

The typical DGS does not need write access to the game assets, so you can have each DGS pod mount the same persistent disk containing read-only assets. To accomplish this, first create the assets disk as described in the Generate an assets disk section, which you’ve already done. Next, create a PersistentVolume in Kubernetes.

In order to include this volume in a Kubernetes DGS pod, you need a PersistentVolumeClaim resource. YAML files for these resources are found in the openarena/k8s/ directory of the solution repository. Once applied, the PersistentVolume detects the asset disk, and the persistentVolumeClaim allows a DGS pod to mount the disk as read-only.

Run the following commands to create the PersistentVolume and the PersistentVolumeClaim:

kubectl apply -f k8s/asset-volume.yaml
kubectl apply -f k8s/asset-volumeclaim.yaml

Note: If you experience connection issues running the kubectl commands, make sure you have kubectl set up correctly, and that you’ve acquired the proper credentials for the new openarena-cluster.

If these commands produce an error saying that the path does not exist, double check the directory you are running the commands from. These commands must be run from the ~/gke-dedicated-game-server/openarena directory.

You can confirm both volume and claim are in Bound status by running the following commands and comparing the output:

kubectl get persistentVolume

Expected output:

asset.volume 50Gi     ROX         Retain        Bound  default /asset.disk.claim assets 3m

Then run:

kubectl get persistentVolumeClaim

Expected output:

asset.disk.claim Bound  asset.volume 50Gi     ROX         assets 2     5s

Configuring the DGS pod

Since scheduling and networking tasks are handled by Kubernetes, and the startup and shutdown time of the DGS containers are relatively negligible, this solution has DGS instances spin up on-demand.

Each DGS instance only lasts the length of a single game match. In addition, it’s implied that each game match has a defined time limit, specified in the OpenArena DGS server configuration file. Once the match is complete, the container successfully exits and if players want to play again they simply request another game.These assumptions simplify a number of pod lifecycle aspects and form the basis of an autoscaling policy discussed in the scaling manager section.

Although this flow isn’t seamless with OpenArena, it’s only because the solution doesn’t fork and change the game client code. In a commercially released game, requesting another match would be made invisible to the user behind previous match result screens and loading times. The code requiring the server change doesn’t represent additional development time as such code is mandatory for handling client reconnections during unforeseen circumstances such as network issues or crashing servers.

For the sake of simplicity, this solution for this lab assumes that the Container Engine nodes have the default network configuration, which assigns each a public IP and allows clients connections.

Managing the dedicated game server process

In commercially produced game servers, all the additional, non-DGS functionality that makes DGSes run as well as containers should be integrated directly into the DGS binaries whenever possible.

As a best practice, the DGS should avoid communicating directly with the matchmaker or scaling manager whenever possible, and instead should expose its state to the Kubernetes API. External processes should read the DGS state from the appropriate Kubernetes endpoints rather than querying the server directly. More information on accessing the Kubernetes API directly can be found here.

Kubernetes resource definitions At first glance, a single process running in a container with a constrained lifetime and defined success criteria would appear to be a use case for Kubernetes jobs, but in practice it’s unnecessary. DGS pods require neither the parallel execution functionality of jobs, nor the ability to guarantee successes by automatically restarting (typically when a session-based DGS dies for some reason, the state is lost, and players simply join another DGS). Due to these factors, scheduling individual Kubernetes pods is preferable for this use case.

In production, DGS pods should be started directly by your matchmaker using the Kubernetes API. For the purposes of this lab, a human-readable YAML file describing the DGS pod resource is included in the solution repository at openarena/k8s/oarena_pod.yaml. When creating configurations for dedicated game server pods, pay close attention to the volume properties to ensure that the asset disk can be mounted as read-only in multiple pods.

Setting up the scaling manager

Note: These commands are run from the repository’s scaling_manager/ directory but assume that you start in the repositories openarena/ directory.

The scaling manager is a simple process that scales the number of virtual machines used as Container Engine nodes based on the current DGS load. In practice, scaling is accomplished using a set of simple scripts that run forever, inspect the total number of DGS pods running and requested, and resize the node pool as necessary. The scripts are packaged in docker container images with the appropriate libraries and Google Cloud SDK.

The docker files creating these images are in the lab’s repository in the scaling_manager/ directory, and the docker image can be created and pushed to gcr.io using the build and push shell script once a number of environment variables are configured.

Build and push the configuration using the following command:

cd ../scaling-manager

These scripts are designed to be be run within a kubernetes deployment , ensuring these processes are restarted in the event of a failure.

Configure the Openarena Scaling Manager Deployment File

An example kubernetes deployment YAML file is provided in the sample solution in the scaling_manager/k8s/openarena-scaling-manager-deployment.yaml directory. This file has to be customized so that the container environment variables it configures use values that describe your lab’s OpenArena cluster.

Find the name of the base instance:

gcloud compute instance-groups managed list

Copy the base instance name. It will be in the fourth column of the output which will look similar to this:


In this next series of commands, replace all of the items in square brackets, including the square brackets themselves, with the appropriate values from your lab. For example, you might replace [GCR_REGION] with “us” and [ZONE] with “us-east1-b”. Both [PROJECT_ID] and [BASE_INSTANCE_ID] will vary for each lab instance:




export GCP_ZONE=[ZONE]

export GCR_REGION=us

export PROJECT_ID=qwiklabs-gcp-6bcf12ce01eb3dc7

export GKE_BASE_INSTANCE_NAME=gke-openarena-cluster-default-pool-08e2eb0b

export GCP_ZONE=us-east1-b


Now run the following to apply the variables you set to the YAML deployment template:

sed -i "s/\[GCR_REGION\]/$GCR_REGION/g" k8s/openarena-scaling-manager-deployment.yaml

sed -i "s/\[PROJECT_ID\]/$PROJECT_ID/g" k8s/openarena-scaling-manager-deployment.yaml

sed -i "s/\[ZONE\]/$GCP_ZONE/g" k8s/openarena-scaling-manager-deployment.yaml

sed -i "s/\gke-openarena-cluster-default-pool-\[REPLACE_ME\]/$GKE_BASE_INSTANCE_NAME/g" k8s/openarena-scaling-manager-deployment.yaml

These commands update the placeholders in the file for the environment variable values listed below with the correct values.

Replacement text Value Notes
[GCR_REGION] gcr.io region such as us, asia, etc. Requires replacement. The region of your gcr.io repository.
[PROJECT_ID] Qwiklabs Project ID, you can copy this from your lab launch page. Requires replacement. The name of your project.
[REPLACE_ME] This is the value you identified in the previous step. You just replace the 8 digit unique id at the end. gke-openarena-cluster-default-pool-[REPLACE_ME]. Requires replacement. Different for every ContainerEngine cluster. The value can be obtained from the output of the gcloud compute instance-groups managed list command.
[ZONE] Set this to the GCP Zone you selected at that start of the lab. This is usually us-east1-b. The name of the GCP zone specified in the gcloud container clusters create command.

The following environment variable values should remain at their defaults.

Replacement text Value Notes
K8S_CLUSTER openarena-cluster The name of the Kubernetes cluster.

You can now add the deployment to your Kubernetes cluster by running:

kubectl apply -f k8s/openarena-scaling-manager-deployment.yaml

Monitor the deployment using kubectl.

kubectl get pods

Wait until you see this report 3/3 nodes ready.

Scaling nodes


To scale nodes, the scaling manager uses the Kubernetes API to look at current node usage and, as needed, resizes the Container Engine cluster’s managed instance group running the underlying virtual machines. Common sticking points for DGS scaling include:

Although the scripts supplied by this solution are basic, their simple design makes it easy to add additional logic. Typical DGSes have well-understood performance characteristics, and by making these into metrics you can determine when to add or remove VM instances. Common scaling metrics are number of DGS instances per CPU, as used in this solution, or available player slots.

Scaling up For this solution, scaling up requires no special handling; simply increasing the number of nodes in the managed instance group is sufficient. For simplicity, this solution uses the limits and requests pod properties in Kubernetes to ‘reserve’ approximately one vCPU and 500MB of memory for each DGS. Since you created the cluster using the n1-highcpu instance family, which has a fixed ratio of 600MB of memory to 1 vCPU, you can safely assume there will be sufficient memory if you schedule one DGS pod per vCPU. Because of this fact, you can scale based on the number of pods in the cluster compared to the number of CPUs in all nodes in the cluster. This ratio determines the remaining resources available, allowing you to add more nodes if the value falls below a threshold. This solution adds nodes if more than 70% of the vCPUs are currently allocated to pods.

In a live, production online game backend, it is recommended that you accurately profile DGS CPU, memory, and network usage and chose your limits and requests pod properties appropriately. For many games, it makes sense to create multiple pod types for different DGS scenarios with different usage profiles, such as game types, specific maps, or number of player slots. Such considerations falls outside the scope of this solution.

Scaling down Scaling down, unlike scaling up, is a multi-step process and one of the major reasons to run a custom, Kubernetes-aware, DGS scaling manager. In this lab, scaling_manger.sh handles the following steps:

Scaling the number of DGS pods In typical production game backends, the matchmaker controls when new DGS instances are added. Since DGS pods are configured to successfully exit when matches complete (refer to the constraints ), no explicit action is necessary to scale down the number of DGS pods. If there are not enough player requests coming into the matchmaker system to generate new matches, the DGS pods will slowly remove themselves from the Kubernetes cluster as matches end, in effect scaling down the number of pods.

Testing the setup

So far, the OpenArena container image has been created and pushed to the container registry, and the DGS Kubernetes cluster has been started. In addition, the game asset disk has been generated and configured for use in Kubernetes, and the scaling manager deployment has been started. Now it’s time to start DGS pods for testing.

Requesting a new DGS instance

In a typical production system, the matchmaker process would directly request an instance using the Kubernetes API when it has appropriate players for a match. For the purposes of testing this solution’s setup, the request for an instance can be made using a simple kubectl command, specifying the openarena_pod.yaml file.

You must first update the file openarena/k8s/openarena-pod.yaml file and replace [GCR_REGION] and [PROJECT_ID] in the containers - image: value. Since the variables are already defined, we can use sed again to do this:

cd ..
sed -i "s/\[GCR_REGION\]/$GCR_REGION/g" openarena/k8s/openarena-pod.yaml
sed -i "s/\[PROJECT_ID\]/$PROJECT_ID/g" openarena/k8s/openarena-pod.yaml

Try this only if the pod errors out at launch.

sed -i "s/\/usr\/share\/games\/openarena\/baseoa/\/usr\/lib\/openarena-server\/baseoa/g"  openarena/k8s/openarena-pod.yaml

Apply the new pod configuration by running:

kubectl apply -f openarena/k8s/openarena-pod.yaml

A few seconds later, the status of the pod can be confirmed using kubectl:

kubectl get pods

Repeat the kubectl get pods command until you get the following output:

openarena.dgs 1/1   Running 0        25s

Connecting to the DGS

After the pod has started, the following commands will identify the game server ip address that you can use to connect to the game instance on port 27461.

export NODE_NAME=$(kubectl get pod openarena.dgs \
    -o jsonpath="{.spec.nodeName}")
export DGS_IP=$(gcloud compute instances list \
    --filter="name=( ${NODE_NAME} )" \

printf "Node Name: $NODE_NAME \nNode IP  : $DGS_IP \nPort         : 27961\n"
printf " launch client with: \nopenarena +connect $DGS_IP +set net_port 27961\n"

Launch the OpenArena client on your own computer once the game has been installed.

Select Multiplayer and then Specify.

Configure the server ip-address and port using those listed by the output of the last command.

Click Connect.

Note: The configuration settings for this solution limit the game server duration to a few minutes, so don’t wait too long before testing the client connection or you might have to start another pod.

Close the OpenArena client.

Testing the scaling manager Since the scaling manager scales the number of VM instances in the Kubernetes cluster based on the number of DGS pods, testing it requires requesting a number of pods over a period of time and checking that the number of nodes scale appropriately.

For testing purposes, a script is provided in the solutions repository which adds four DGS pods per minute for 5 minutes:


In your own environment, make sure to set the match length in the server configuration file to an appropriate limit so that the pods eventually exit and you can see the nodes scale back down! For this lab, the server configuration file places a five minute time limit on the match, and can be used as an example. It’s located in the repo at openarena/single-match.cfg, and is used by default.

Open the Google Cloud Platform Console and click Kubernetes Engine, then click Workloads.

You will see a sequence of workloads startup called openarena-dgs.s, openarena-dgs.2 up to openarena-dgs.15. Because there are a limited the number of vCPUs in this cluster, many of the test containers will initially show an error state with a status of “Unschedulable”. As the startup load in each container reduces, some of the later game server containers will successfully start up. All containers will start after about 10 minutes.

If you explore any of the failed instances you will see that the reason for the failure is a lack of CPU resources. If you explore any of the running instances you will see that the game server has initialized on those instances.

Awwvision: Cloud Vision API from a Kubernetes Cluster


The Awwvision lab uses Kubernetes and Cloud Vision API to demonstrate how to use the Vision API to classify (label) images from Reddit’s /r/aww subreddit and display the labelled results in a web app.

Awwvision has three components:

  1. A simple Redis instance.
  2. A web app that displays the labels and associated images.
  3. A worker that handles scraping Reddit for images and classifying them using the Vision API. Cloud Pub/Sub is used to coordinate tasks between multiple worker instances.

Create a Kubernetes Engine cluster

In this lab you will use gcloud, Google Cloud Platform’s command-line tool, to set up a Kubernetes Engine cluster. You can specify as many nodes as you want, but you need at least one. The cloud platform scope is used to allow access to the Pub/Sub and Vision APIs.

In Cloud Shell, run the following to create a cluster in the us-central1-f zone:

gcloud config set compute/zone us-central1-f

Then start up the cluster by running:

gcloud container clusters create awwvision \
    --num-nodes 2 \
    --scopes cloud-platform

Run the following to use the container’s credentials:

gcloud container clusters get-credentials awwvision

Verify that everything is working using the kubectl command-line tool:

kubectl cluster-info

Get the Sample

Now add sample data to your project by running:

git clone https://github.com/GoogleCloudPlatform/cloud-vision

Deploy the sample

In Cloud Shell, change to the python/awwvision directory in the cloned cloud-vision repo:

cd cloud-vision/python/awwvision

Once in the awwvision directory, run make all to build and deploy everything:

make all

As part of the process, Docker images will be built and uploaded to the Google Container Registry private container registry. In addition, yaml files will be generated from templates, filled in with information specific to your project, and used to deploy the redis, webapp, and worker Kubernetes resources for the lab.

Check the Kubernetes resources on the cluster

After you’ve deployed, check that the Kubernetes resources are up and running.

First, list the pods by running:

kubectl get pods

You should see something like the following, though your pod names will be different. Make sure all of your pods have a Running before executing the next command.

NAME                     READY     STATUS    RESTARTS   AGE
awwvision-webapp-vwmr1   1/1       Running   0          1m
awwvision-worker-oz6xn   1/1       Running   0          1m
awwvision-worker-qc0b0   1/1       Running   0          1m
awwvision-worker-xpe53   1/1       Running   0          1m
redis-master-rpap8       1/1       Running   0          2m

Next, list the deployments by running:

kubectl get deployments -o wide

You can see the number of replicas specified for each, and the images used.

NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS         IMAGES                                SELECTOR
awwvision-webapp   1         1         1            1           1m        awwvision-webapp   gcr.io/your-project/awwvision-webapp   app=awwvision,role=frontend
awwvision-worker   3         3         3            3           1m        awwvision-worker   gcr.io/your-project/awwvision-worker   app=awwvision,role=worker
redis-master       1         1         1            1           1m        redis-master       redis                                 app=redis,role=master

Once deployed, get the external IP address of the webapp service by running:

kubectl get svc awwvision-webapp

It may take a few minutes for the assigned external IP to be listed in the output. You should see something like the following, though your IPs will be different.

NAME               CLUSTER_IP      EXTERNAL_IP    PORT(S)   SELECTOR                      AGE
awwvision-webapp   80/TCP    app=awwvision,role=frontend   13m

Visit your new web app and start its crawler

Copy and paste the external IP of the awwvision-webapp service into a new browser to open the webapp, then click Start the Crawler button.

Next, click go back and you should start to see images from the /r/aww subreddit classified by the labels provided by the Vision API. You will see some of the images classified multiple times, when multiple labels are detected for them. (You can reload in a bit, in case you brought up the page before the crawler was finished).

Running a MongoDB Database in Kubernetes with StatefulSets


Kubernetes is an open source container orchestration tool that handles the complexities of running containerized applications. You can run Kubernetes applications with Kubernetes Engine—a GCP computing service that offers many different customizations and integrations. In this lab, you will get some practical experience with Kubernetes by learning how to set up a MongoDB database with a StatefulSet. Running a stateful application (a database) on a stateless service (container) may sound contradictory. However, after getting hands-on practice with this lab you will quickly see how that’s not the case. In fact, by using a few open-source tools you will see how Kubernetes and stateless services can go hand-in-hand.

What you’ll learn In this lab, you will learn the following:

Set a Compute Zone

Throughout this lab, we will be using the gcloud command line tool to provision our services. Before we can create our Kubernetes cluster, we will need to set a compute zone so that the virtual machines in our cluster are all created in the same region. We can do this using the gcloud config set command—run the following in your cloud shell to set your zone to us-central1-f:

gcloud config set compute/zone us-central1-f

? Note: More information about regions and zones is available here.

Create a new Cluster

Now that our zone is set, we will create a new cluster of containers. Run the following command to instantiate a cluster named hello-world:

gcloud container clusters create hello-world

This command creates a new cluster with three nodes, or virtual machines (the default). You can configure this command with additional flags to change the number of nodes, the default permissions, and other variables. See the documentation for more details.

Launching the cluster may take a few minutes. Once it’s up, you should receive a similar output:

NAME         Location       MATER_VERSION   MASTER_IP       ...
hello-world  us-central1-f  1.9.7-gke.3  ...

Setting up

Now that we have our cluster up and running, it’s time to integrate it with MongoDB. We will be using a replica set so that our data is highly available and redundant—a must for running production applications. To get set up, we need to do the following:

git clone https://github.com/thesandlord/mongo-k8s-sidecar.git

Once it’s cloned, navigate to the StatefulSet directory with the following command:

cd ./mongo-k8s-sidecar/example/StatefulSet/

Once you have verified that the files have been downloaded and that you’re in the right directory, let’s go ahead and create a Kubernetes StorageClass.

Create the StorageClass

A StorageClass tells Kubernetes what kind of storage you want to use for database nodes. On the Google Cloud Platform, you have a couple of storage choices: SSDs and hard disks.

If you take a look inside the StatefulSet directory (you can do this by running the ls command), you will see SSD and HDD configuration files for both Azure and GCP. Run the following command to take a look at the googlecloud_ssd.yaml file:

cat googlecloud_ssd.yaml


kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
  name: fast
provisioner: kubernetes.io/gce-pd
  type: pd-ssd

This configuration creates a new StorageClass called “fast” that is backed by SSD volumes. Run the following command to deploy the StorageClass:

kubectl apply -f googlecloud_ssd.yaml

Now that our StorageClass is configured, our StatefulSet can now request a volume that will automatically be created.

Deploying the Headless Service and StatefulSet

Find and inspect the files

Before we dive into what headless service and StatefulSets are, let’s open up the configuration file (mongo-statefulset.yaml) where they are both housed in.

cat mongo-statefulset.yaml

You should receive the following output (without the pointers to the Headless Service and StatefulSet content):

apiVersion: v1   <-----------   Headless Service configuration
kind: Service
  name: mongo
    name: mongo
  - port: 27017
    targetPort: 27017
  clusterIP: None
    role: mongo
apiVersion: apps/v1beta1    <------- StatefulSet configuration
kind: StatefulSet
  name: mongo
  serviceName: "mongo"
  replicas: 3
        role: mongo
        environment: test
      terminationGracePeriodSeconds: 10
        - name: mongo
          image: mongo
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
            - containerPort: 27017
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"
  - metadata:
      name: mongo-persistent-storage
        volume.beta.kubernetes.io/storage-class: "fast"
      accessModes: [ "ReadWriteOnce" ]
          storage: 100Gi

Headless service: overview The first section of mongo-statefulset.yaml refers to a headless service. In Kubernetes terms, a service describes policies or rules for accessing specific pods. In brief, a headless service is one that doesn’t prescribe load balancing. When combined with StatefulSets, this will give us individual DNSs to access our pods, and in turn a way to connect to all of our MongoDB nodes individually. In the yaml file, you can make sure that the service is headless by verifying that the clusterIP field is set to None.

StatefulSet: overview The StatefulSet configuration is the second section of mongo-statefulset.yaml. This is the bread and butter of the application: it’s the workload that runs MongoDB and what orchestrates your Kubernetes resources. Referencing the yaml file, we see that the first section describes the StatefulSet object. Then, we move into the Metadata section, where labels and the number of replicas are specified.

Next comes the pod spec. The terminationGracePeriodSeconds is used to gracefully shutdown the pod when you scale down the number of replicas. Then the configurations for the two containers are shown. The first one runs MongoDB with command line flags that configure the replica set name. It also mounts the persistent storage volume to /data/db: the location where MongoDB saves its data. The second container runs the sidecar. This sidecar container will configure the MongoDB replica set automatically. As mentioned earlier, a “sidecar” is a helper container that helps the main container run its jobs and tasks.

Finally, there is the volumeClaimTemplates. This is what talks to the StorageClass we created before to provision the volume. It provisions a 100 GB disk for each MongoDB replica.

Deploy Headless Service and the StatefulSet Now that we have a basic understanding of what a headless service and StatefulSet are, let’s go ahead and deploy them. Since the two are packaged in mongo-statefulset.yaml, we can run the following comand to run both of them:

kubectl apply -f mongo-statefulset.yaml

You should receive the following output:

service "mongo" created
statefulset "mongo" created

Connect to the MongoDB Replica Set

Now that we have a cluster running and our replica set deployed, let’s go ahead and connect to it.

Wait for the MongoDB replica set to be fully deployed

Kubernetes StatefulSets deploys each pod sequentially. It waits for the MongoDB replica set member to fully boot up and create the backing disk before starting the next member. Run the following command to view and confirm that all three members are up:

kubectl get statefulset

Output - all three members are up.

mongo     3         3         3m

Initiating and Viewing the MongoDB replica set

At this point, you should have three pods created in your cluster. These correspond to the three nodes in your MongoDB replica set. Run this command to view:

kubectl get pods


mongo-0     2/2       Running   0          3m
mongo-1     2/2       Running   0          3m
mongo-2     2/2       Running   0          3m

Wait for all three members to be created before moving on.

Connect to the first replica set member:

kubectl exec -ti mongo-0 mongo

You now have a REPL environment connected to the MongoDB .

Let’s instantiate the replica set with a default configuration by running the rs.initiate() command:


Print the replica set configuration; run the rs.conf() command:


This outputs the details for the current member of replica set rs0. In this lab you see only one member. To get details of all members you need to expose the replica set through additional services like nodeport or load balancer.

rs0:OTHER> rs.conf()
        "_id" : "rs0",
        "version" : 1,
        "protocolVersion" : NumberLong(1),
        "writeConcernMajorityJournalDefault" : true,
        "members" : [
                        "_id" : 0,
                        "host" : "localhost:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {
                        "slaveDelay" : NumberLong(0),
                        "votes" : 1
        "settings" : {
                "chainingAllowed" : true,
                "heartbeatIntervalMillis" : 2000,
                "heartbeatTimeoutSecs" : 10,
                "electionTimeoutMillis" : 10000,
                "catchUpTimeoutMillis" : -1,
                "catchUpTakeoverDelayMillis" : 30000,
                "getLastErrorModes" : {
                "getLastErrorDefaults" : {
                        "w" : 1,
                        "wtimeout" : 0
                "replicaSetId" : ObjectId("5c526b6501fa2d29fc65c48c")

Type “exit” and press enter to quit the REPL.

Scaling the MongoDB replica set

A big advantage of Kubernetes and StatefulSets is that you can scale the number of MongoDB Replicas up and down with a single command!

To scale up the number of replica set members from 3 to 5, run this command:

kubectl scale --replicas=5 statefulset mongo

In a few minutes, there will be 5 MongoDB pods. Run this command to view them:

kubectl get pods

To scale down the number of replica set members from 5 back to 3, run this command:

kubectl scale --replicas=3 statefulset mongo

In a few seconds, there are 3 MongoDB pods. Run this command to view:

kubectl get pods

Your output should look like this:

mongo-0   2/2       Running   0          41m
mongo-1   2/2       Running   0          39m
mongo-2   2/2       Running   0          37m

Using the MongoDB replica set

Each pod in a StatefulSet backed by a Headless Service will have a stable DNS name. The template follows this format: .

This means the DNS names for the MongoDB replica set are:


You can use these names directly in the connection string URI of your app.

Using a database is outside the scope of this lab, however for this case, the connection string URI would be:


Clean up

Because you are working in Qwiklabs, when you end the lab all your resources and your project will be cleaned up and discarded on your behalf. But we want to show you how to clean up resources yourself to save on cost and to be a good cloud citizen when you are in your own environment.

To clean up the deployed resources, run the following commands to delete the StatefulSet, Headless Service, and the provisioned volumes.

Delete the StatefulSet:

kubectl delete statefulset mongo

Delete the service:

kubectl delete svc mongo

Delete the volumes:

kubectl delete pvc -l role=mongo

Finally, you can delete the test cluster:

gcloud container clusters delete "hello-world"

Press Y then Enter to continue deleting the test cluster.

Kubeflow End to End


Kubeflow overview

Kubeflow is a machine learning toolkit for Kubernetes. The project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable, and scalable. The goal is to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures.

A Kubeflow deployment is:

Kubeflow will let you organize loosely-coupled microservices as a single unit and deploy them to a variety of locations, whether that’s a laptop or the cloud. This codelab will walk you through creating your own Kubeflow deployment.

What you’ll build

In this lab you’re going to build a web app that summarizes GitHub issues using a trained model. Upon completion, your infrastructure will contain:

A Kubernetes Engine cluster with standard Kubeflow and Seldon Core installations. A training job that uses Tensorflow to generate a Keras model. A serving container that provides predictions. A UI that uses the trained model to provide summarizations for GitHub issues.

What you’ll learn

What you’ll need

Set up

Enable Boost Mode

In the Cloud Shell window, click on the Settings icon at the far right. Select Enable Boost Mode and then select “Restart Cloud Shell in Boost Mode”. This will provision a larger instance for your Cloud Shell session, resulting in speedier Docker builds.

Enable Boost Mode

Download the project files Set the correct version and an environment variable:

export KUBEFLOW_TAG=0.4.0-rc.2

Once your Cloud Shell has been provisioned, run the following commands to download and unpack an archive of the Kubeflow examples repo, which contains all of the official Kubeflow examples:

git clone https://github.com/kubeflow/examples.git
cd examples
git checkout v${KUBEFLOW_TAG}

Set your GitHub token

This lab involves the use of many different files obtained from public repos on GitHub. To prevent rate-limiting, especially at events where a large number of anonymized requests are sent to the GitHub APIs, you will set up an access token with no permissions. This is simply to authorize you as an individual rather than anonymous user.

Sign into your GitHub account in a new tab, then navigate to: https://github.com/settings/tokens.

Click Generate a new token, select no permissions.

Click on the Copy icon.

Back in Cloud Shell, set the GITHUB_TOKEN environment variable, replacing <token> with your GitHub token:

export GITHUB_TOKEN=<token>
export GITHUB_TOKEN=a10a849c8ef485571de2d9d429f6796f2e3fa3e9

Install pyyaml

Ensure that pyyaml is installed by running:

pip install --user pyyaml

Install ksonnet

Set the correct version as an environment variable:

export KS_VER=0.13.1
export KS_BIN=ks_${KS_VER}_linux_amd64

Download and unpack ksonnet, then add it to your $PATH:

wget -O /tmp/${KS_BIN}.tar.gz https://github.com/ksonnet/ksonnet/releases/download/v${KS_VER}/${KS_BIN}.tar.gz
mkdir -p ${HOME}/bin
tar -xvf /tmp/${KS_BIN}.tar.gz -C ${HOME}/bin
export PATH=$PATH:${HOME}/bin/${KS_BIN}

Note: To familiarize yourself with ksonnet concepts, see this diagram.

Install kfctl

Download and unpack kfctl, the Kubernetes command-line tool, then add it to your $PATH:

wget -P /tmp https://github.com/kubeflow/kubeflow/archive/v${KUBEFLOW_TAG}.tar.gz
mkdir -p ${HOME}/src
tar -xvf /tmp/v${KUBEFLOW_TAG}.tar.gz -C ${HOME}/src
cd ${HOME}/src/kubeflow-${KUBEFLOW_TAG}/scripts
ln -s kfctl.sh kfctl
export PATH=$PATH:${HOME}/src/kubeflow-${KUBEFLOW_TAG}/scripts
cd ${HOME}

kubectl allows you to run commands against Kubernetes clusters, deploy applications, inspect and manage cluster resources, and view logs.

Set your GCP project ID

Store your Project ID as an environment variable:

export PROJECT_ID=<gcp_project_id>
export PROJECT_ID=qwiklabs-gcp-86fddff50e06d8bf

Activate the latest scopes in Kubernetes Engine:

export ZONE=us-central1-a
gcloud config set project ${PROJECT_ID}
gcloud config set compute/zone ${ZONE}
gcloud config set container/new_scopes_behavior true

Authorize Docker

Allow Docker access to your project’s Container Registry:

gcloud auth configure-docker

Enter in Y when promted to update the Docker configuration file.

Create a service account

Create a service account with read/write access to storage buckets:

export SERVICE_ACCOUNT=user-gcp-sa
export SERVICE_ACCOUNT_EMAIL=${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
gcloud iam service-accounts create ${SERVICE_ACCOUNT} \
  --display-name "GCP Service Account for use with kubeflow examples"
gcloud projects add-iam-policy-binding ${PROJECT_ID} --member \
  serviceAccount:${SERVICE_ACCOUNT_EMAIL} \

Generate a credentials file for upload to the cluster:

export KEY_FILE=${HOME}/secrets/${SERVICE_ACCOUNT_EMAIL}.json
gcloud iam service-accounts keys create ${KEY_FILE} \
  --iam-account ${SERVICE_ACCOUNT_EMAIL}

Create a storage bucket

Create a Cloud Storage bucket for storing your trained model and issue the mb (make bucket) command:

export BUCKET_NAME=kubeflow-${PROJECT_ID}
gsutil mb -c regional -l us-central1 gs://${BUCKET_NAME}

Click Check my progress to verify the objective.

Create a cluster

Create a managed Kubernetes cluster on Kubernetes Engine by running:

kfctl init kubeflow-qwiklab --platform gcp --project ${PROJECT_ID}

(Enabling services can take several minutes.)

cd kubeflow-qwiklab
kfctl generate platform
sed -i 's/n1-standard-8/n1-standard-4/g' gcp_config/cluster.jinja
kfctl apply platform

Cluster creation will take a few minutes to complete.

Ignore the error about setting CLIENT_ID. In the interest of time, you are skipping this step.

To verify the connection, run the following command:

kubectl cluster-info

Verify that this IP address matches the IP address corresponding to the Endpoint in your Google Cloud Platform Console by comparing the Kubernetes master IP is the same as the Master_IP address in the previous step.

Upload service account credentials

kubectl create secret generic user-gcp-sa \

Install Kubeflow

Next, install Kubeflow:

kfctl generate k8s
kfctl apply k8s

Installing Kubeflow will take several minutes.

ksonnet is a templating framework which allows you to utilize common object definitions and customize them to your environment. You’ll begin by referencing Kubeflow templates and apply environment-specific parameters. Once manifests have been generated specifically for your cluster, they can be applied like any other Kubernetes object using kubectl.

Add Seldon to the default installation

Run the following to install Seldon:

cd ${HOME}/kubeflow-qwiklab/ks_app
ks generate seldon seldon
ks apply default -c seldon

Your cluster now contains a Kubernetes installation with Seldon, which has the following components:

You can view the components by running:

kubectl get pods

Note: Image pulls can take a while. You can expect pods to remain in ContainerCreating status for a few minutes.

You should see output similar to this:


Train a model

In this section, you will create a component that trains a model.

Set the component parameters:

cd ${HOME}/kubeflow-qwiklab/ks_app
ks generate tf-job-simple-v1beta1 tfjob --name tfjob-issue-summarization
cp ${HOME}/examples/github_issue_summarization/ks_app/components/tfjob.jsonnet components/
ks param set tfjob gcpSecretName "user-gcp-sa"
ks param set tfjob gcpSecretFile "user-gcp-sa.json"
ks param set tfjob image "gcr.io/kubeflow-examples/tf-job-issue-summarization:v20180629-v0.1-2-g98ed4b4-dirty-182929"
ks param set tfjob input_data "gs://kubeflow-examples/github-issue-summarization-data/github_issues_sample.csv"
ks param set tfjob input_data_gcs_bucket "kubeflow-examples"
ks param set tfjob input_data_gcs_path "github-issue-summarization-data/github-issues.zip"
ks param set tfjob num_epochs "7"
ks param set tfjob output_model "/tmp/model.h5"
ks param set tfjob output_model_gcs_bucket "${BUCKET_NAME}"
ks param set tfjob output_model_gcs_path "github-issue-summarization-data"
ks param set tfjob sample_size "100000"

The training component tfjob is now configured to use a pre-built container image.

Launch training

Ensure that you are in the proper working directory by running the following command:

cd ${HOME}/kubeflow-qwiklab/ks_app

Apply the component manifests to the cluster:

ks apply default -c tfjob

View the running job

View the resulting pods:

kubectl get pod -l tf_job_name=tfjob-issue-summarization

Your cluster state should look similar to this:

View the running job

It can take a few minutes to pull the image and start the container. Re-run the command until you see the status is Running.

Once tfjob-issue-summarization-master-0 is running, tail the logs:

kubectl logs -f tfjob-issue-summarization-master-0

Inside the pod, you will see the download of source data (github-issues.zip) before training begins. Continue tailing the logs until the pod exits on its own and you find yourself back at the command prompt.

Print the logs for a container in a pod or specified resource. If the pod has only one container, the container name is optional.

To verify that training completed successfully, check to make sure all three model files were uploaded to your Cloud Storage bucket:

gsutil ls gs://${BUCKET_NAME}/github-issue-summarization-data

You should see something like this:


Serve the trained model

Next you will create a component that serves a trained model.

Set component parameters:

export SERVING_IMAGE=gcr.io/kubeflow-examples/issue-summarization-model:v20180718-g98ed4b4-qwiklab

Create the serving component

The serving component is configured to run a pre-built image. Using a Seldon ksonnet template, generate the serving component.

Navigate back to the ksonnet app directory for Kubeflow, and issue the following command:

cd ${HOME}/kubeflow-qwiklab/ks_app
ks generate seldon-serve-simple-v1alpha2 issue-summarization-model \
  --name=issue-summarization \
  --image=${SERVING_IMAGE} \

Launch serving

Apply the component manifests to the cluster:

ks apply default -c issue-summarization-model

View the running pods

You will see several new pods appear:

kubectl get pods -l seldon-deployment-id=issue-summarization


several new pods appear

Your cluster state should look similar to this:

cluster state

Wait a minute or two and re-run the previous command. Once the pods are running, run the following command to view the logs for one of the serving containers to verify that it is running on port 9000:

kubectl logs \
  $(kubectl get pods \
    -lseldon-deployment-id=issue-summarization \
    -o=jsonpath='{.items[0].metadata.name}') \

You should see a similar output:

 * Running on (Press CTRL+C to quit)

Add a UI

In this section, you will create a component that provides browser access to the serving component.

Set parameter values

cd ${HOME}/kubeflow-qwiklab/ks_app
ks generate deployed-service ui \
  --name issue-summarization-ui \
  --image gcr.io/kubeflow-examples/issue-summarization-ui:v20180629-v0.1-2-g98ed4b4-dirty-182929

cp ${HOME}/examples/github_issue_summarization/ks_app/components/ui.jsonnet components/

ks param set ui githubToken ${GITHUB_TOKEN}
ks param set ui modelUrl "http://issue-summarization.kubeflow.svc.cluster.local:8000/api/v0.1/predictions"

The UI component is now configured to use a pre-built container image which is made available in Container Registry (gcr.io).

Note: If you would prefer to generate your own image instead, continue with the (Optional) Create the UI Image step. Otherwise, continue with Launch the UI.

(Optional) Create the UI image

Image creation can take 5-10 minutes. This step is optional. Alternatively, skip directly to the Launch the UI section below.

Switch to the docker directory and build the image for the UI:

cd ${HOME}/examples/github_issue_summarization/docker
docker build -t gcr.io/${PROJECT_ID}/issue-summarization-ui:latest .

After the image has been successfully built, store it in Container Registry:

docker push gcr.io/${PROJECT_ID}/issue-summarization-ui:latest

Update the component parameter with a link that points to the custom image:

cd ${HOME}/kubeflow-qwiklab/ks_app
ks param set ui image gcr.io/${PROJECT_ID}/issue-summarization-ui:latest

Launch the UI

Apply the component manifests to the cluster:

ks apply default -c ui

You should see an additional pod with the status ContainerCreating:

kubectl get pods -l app=issue-summarization-ui

Launch the UI

View the UI

To view the UI, open a port to the ambassador service:

kubectl port-forward svc/ambassador 8080:80

In Cloud Shell, click on the Web Preview button and select “Preview on port 8080.”

Preview on port 8080

This will open a new browser tab that shows the Kubeflow Central Dashboard. Add the text “issue-summarization/” to the end of the URL and press Enter (don’t forget the trailing slash).

You should see something like this:

Preview UI

Click the Populate Random Issue button to fill in the large text box with a random issue summary. Then click the Generate Title button to view the machine generated title produced by your trained model. Click the button a couple of times to give yourself some more data to look at in the next step.

View serving container logs

In Cloud Shell, tail the logs of one of the serving containers to verify that it is receiving a request from the UI and providing a prediction in response:

kubectl logs -f \
  $(kubectl get pods \
    -lseldon-deployment-id=issue-summarization \
    -o=jsonpath='{.items[0].metadata.name}') \

Back in the UI, press the Generate Title button a few times to view the POST request in Cloud Shell. Since there are two serving containers, you might need to try a few times before you see the log entry.

google3528628_student@cloudshell:~/kubeflow-qwiklab/ks_app (qwiklabs-gcp-86fddff50e06d8bf)$ kubectl logs -f \
>   $(kubectl get pods \
>     -lseldon-deployment-id=issue-summarization \
>     -o=jsonpath='{.items[0].metadata.name}') \
>   issue-summarization
Using TensorFlow backend.
body_pp file body_pp.dpkl
title_pp file title_pp.dpkl
model file seq2seq_model_tutorial.h5
2019-05-29 23:57:43.232121: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this
 TensorFlow binary was not compiled to use: AVX2 FMA
 * Running on (Press CTRL+C to quit) - - [30/May/2019 00:07:57] "POST /predict HTTP/1.1" 200 - - - [30/May/2019 00:08:35] "POST /predict HTTP/1.1" 200 - - - [30/May/2019 00:09:01] "POST /predict HTTP/1.1" 200 -

Clean up

Remove GitHub token Navigate to https://github.com/settings/tokens and remove the token you generated for this lab.

Delete the Cluster

In Cloud Shell, run the following command to delete the cluster:

gcloud container clusters delete kubeflow-qwiklab –zone=us-central1-a

Sample output:

The following clusters will be deleted.
 - [kubeflow-qwiklab] in [us-central1-a]

Do you want to continue (Y/n)?  Y

Deleting cluster kubeflow-qwiklab...done.
Deleted [https://container.googleapis.com/v1/projects/qwiklabs-gcp-419184863edcf031/zones/us-central1-a/clusters/kubeflow-qwiklab]

Deploy a Web App on GKE with HTTPS Redirect using Lets Encrypt


GKE does not provide a managed HTTPS offering, so it can be a bit daunting trying to take on the task of obtaining a valid TLS certificate without prior experience. You will need to find a Certificate Authority (CA) to provide a browser-trusted certificate and you need a way to manage those certificates.

With Let’s Encrypt, you have access to a free, automated, and open certificate authority (CA), run for the public’s benefit. Let’s Encrypt provides a browser-trusted certificate for your web services. In combination with cert-manager, a Kubernetes add-on, the management and issuance of TLS certificates from Let’s Encrypt will be completely automated.

Since GKE also lacks built-in HTTP to HTTPs redirect for Google Cloud Load Balancers (GCLB), an NGINX ingress will be deployed to handle HTTP to HTTPs redirect.

What you will build

In this lab, you’re going to deploy a containerized web app in a GKE cluster with HTTPS using a browser-trusted TLS certificate and NGINX to route all HTTP traffic to HTTPS. Google Cloud Endpoints is used for its ability to dynamically provision DNS entries under cloud.goog DNS domain.

What you’ll learn

In this lab you’ll learn how to do the following:

What you’ll need

Download the source code

In Cloud Shell, download the source code for the lab:

wget https://storage.googleapis.com/vwebb-codelabs/gke-tls-qwik/gke-tls-lab.tar.gz

Unpack the file to your local system and navigate to the source code directory:

tar zxfv gke-tls-lab.tar.gz

cd gke-tls-lab

Configure Cloud Endpoints

Allocate a Static IP

First, allocate a static IP in the GCP region our cluster resides using the following command:

gcloud compute addresses create endpoints-ip --region us-central1

Verify an address is allocated:

gcloud compute addresses list

The IP address in the output is the allocated IP address. Record this IP address as you will use it throughout the lab.

Launch the Cloud Shell code editor

In this lab you’ll view and edit files. You can use the shell editors that are installed on Cloud Shell, such as nano or vim, or use the Cloud Shell code editor. This lab uses the Cloud Shell code editor.

Launch the Cloud Shell code editor by clicking the pencil icon:

Update openapi.yaml.

In the left pane in the code editor, navigate to gke-tls-lab/openapi.yaml.

Replace [MY-STATIC-IP] with the allocated IP address.

In the code, replace all instances of [MY-PROJECT] with your GCP Project ID and save openapi.yaml.

Deploy the Cloud Endpoints

Run the following command to deploy to Cloud Endpoints:

gcloud endpoints services deploy openapi.yaml

Create a Kubernetes Engine Cluster

In the command line, create a cluster by running the following command:

gcloud container clusters create cl-cluster --zone us-central1-f

It may take a few minutes to complete.

Authentication credentials for the cluster

You need authentication credentials to interact with the cluster.

Get authentication credentials:

gcloud container clusters get-credentials cl-cluster --zone us-central1-f

Set up Role-Based Access Control

To be able to deploy to the cluster, you need the proper permissions.

Assign yourself the cluster-admin role by running the following command:

kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account)

Install Helm

Helm is a Kubernetes package-manager that helps you manage Kubernetes applications that we will use in the lab to install our NGINX ingress and Let’s Encrypt.

Download and install Helm client:

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > get_helm.sh

chmod 700 get_helm.sh


Next, install the Helm server-side components (Tiller) on your GKE cluster by running the following command:

kubectl create serviceaccount -n kube-system tiller

kubectl create clusterrolebinding tiller-binding \
    --clusterrole=cluster-admin \
    --serviceaccount kube-system:tiller

helm init --service-account tiller

Now update Helm’s chart repositories:

helm repo update

Install NGINX Ingress

You will deploy an NGINX ingress using Helm to handle our HTTP to HTTPS redirect when configure our web app for HTTPS to ensure the user always has a secure connection to our app.

Install the NGINX ingress using Helm by running the following command. Replace [MY-STATIC-IP] with the allocated IP address you recorded in a previous section.

helm install stable/nginx-ingress --set controller.service.loadBalancerIP="[MY-STATIC-IP]",rbac.create=true

helm install stable/nginx-ingress --set controller.service.loadBalancerIP="",rbac.create=true

You can retrieve the allocated IP address by running:

gcloud compute addresses list

Deploy “Hello World” App

Now that the cluster is created, you can deploy the containerized “hello world” web app.

Update configmap.yaml

The file configmap.yaml contains the environment variable used by Cloud Endpoints.

In the code editor, navigate to configmap.yaml and replace [MY-PROJECT] with your GCP Project ID.

Save the file.

Deploy web app to cluster

Run the following commands to deploy:

kubectl apply -f configmap.yaml

kubectl apply -f deployment.yaml

Now, we need to expose our deployment. We are going to expose our web app by first attaching a NodePort service by running the following command:

kubectl apply -f service.yaml

With our service created, we can now deploy an ingress to create a Google Cloud Load Balancer to reach our service externally. The ingress will include the following annotation:

kubernetes.io/ingress.class: nginx

This annotation will tell Kubernetes not to create a Google Cloud Load Balancer and to use our NGINX ingress controller instead.

Update ingress.yaml

In the code editor, navigate to ingress.yaml and replace all instances [MY-PROJECT] with your GCP Project ID.

Save the file

Apply the ingress and test

In the command line, apply the ingress:

kubectl apply -f ingress.yaml

Note: It might take 5-10 minutes for the ingress to be properly provisioned.

Test the application. Open a new browser window and connect to the address below, replacing [MY-PROJECT] with your GCP Project ID:


You have successfully deployed the web app when you see the “Hello, world” message, version, and hostname displayed in the browser.

Now to configure the web app for HTTPS!


We will install cert-manager and configure Let’s Encrypt as our certificate issuer to obtain a browser-trusted TLS certificates that can be used to secure our application with HTTPS, and configure our existing esp-ingress to handle only HTTPS requests.

Set up Let’s Encrypt

We will use Helm to install cert-manager using the following command:

helm install --name cert-manager --version v0.3.2 \
    --namespace kube-system stable/cert-manager

Run the following command to set your email address in a variable:

export EMAIL=ahmet@example.com

Now we will deploy Let’s Encrypter issuer to issue TLS certificates. Deploy the Issuer manifest using the following command:

cat letsencrypt-issuer.yaml | sed -e "s/email: ''/email: $EMAIL/g" | kubectl apply -f-

Reconfigure ingress for HTTPS

We will now configure our existing esp-ingress with TLS. This is done by deploying the ingress modified version of our ingress.yaml, which is ingress-tls.yaml.

The file, ingress-tls.yaml modifies the ingress with the following additional annotations and modifications to the spec key:



kubernetes.io/tls-acme: "true"

certmanager.k8s.io/cluster-issuer: letsencrypt-prod

ingress.kubernetes.io/ssl-redirect: "true"



- hosts:

- api.endpoints.[MY-PROJECT].cloud.goog

secretName: esp-tls

The annotations will tell cert-manager to begin automating the process of acquiring the required TLS certificate from Let's Encrypt, and will tell our NGINX ingress to redirect all HTTP traffic to HTTPS. The spec will include a "tls" key containing our hosts and secretName.
Update ingress-tls.yaml

Navigate to ingress-tls.yaml. Rename all instances of [MY-PROJECT] with your GCP Project ID and save the file.

Apply the updates

In the command line, apply the change to esp-ingress:

kubectl apply -f ingress-tls.yaml

Once again, you can check the status by running the following command:

kubectl describe ingress esp-ingress

You should see output similar to the following:

  Type    Reason             Age                From                      Message
  ----    ------             ----               ----                      -------
  Normal  CREATE             10m                nginx-ingress-controller  Ingress default/esp-ingress
  Normal  UPDATE             23s (x2 over 10m)  nginx-ingress-controller  Ingress default/esp-ingress

Note: It might take a few minutes for the ingress to be properly provisioned.

Test the application by connecting to the address below, replacing [MY-PROJECT] with your GCP Project ID:


Now, look at the address bar and you will notice that you are now utilizing a secure, encrypted HTTPS connection with a browser-trusted certificate!