Setting up the private Kubernetes cluster on GKE. The working one.

Kubernetes is everywhere. It is a very good choice when few Docker containers on single server is not enough. It’s an abstraction layer between application and infrastructure. And — well, it’s complicated.

If you want to setup own kubernetes cluster you have to learn a lot of new things. Etcd cluster, certificates, several networking options — hard to setup and probably harder to maintain. Maybe it’s a good idea to check one of managed solutions?

Google Kubernetes Engine

It’s a natural first choice. Who has more experience with running large scale Kubernetes clusters than Google? :). It allows to run new cluster in minutes and eliminates the need of maintaining and operating the cluster.

Let’s play with it. I assume you already have an account on Google Cloud Platform and gcloud CLI utility installed.

Using the console

It’s a best practice to setup infrastructure using some Infrastructure as a Code tool like for example terraform. You can also use gcloud to do it. But I recommend you to start with the graphical console on GCP website to see all options you have.

On the menu choose Kubernetes Engine > Kubernetes Clusters and click CREATE CLUSTER.

On the top you can assign the cluster name, enter some nice description and choose the location which determines if your nodes will live in single availability zone or will be spread over multiple zones within region. Depending on the choice you can select the zone or region.

Then you can choose version of kubernetes, the node size, image and number of nodes per availability zone. Note — if you have choosen Regional above you will have at least 3 nodes. You also see a nice recap of your resources. In this case I changed the default settings and chose newest version of kubernetes.

On the bottom of the page is an interesting button:

Lets’ check what hides there. Wow, a lot of options. You can label your resources, add additional zones or even enable alpha features of the cluster. And finally an Autoscaling where GKE can dynamically fire new nodes and shutdown redundant ones.

Here you can choose to run your cluster on preemptible nodes — definitely a good choice for test purposes — your cluster doesn't have to be online 24/7 and the costs are significantly lower.

To be honest this option puzzled me most: do I want a private cluster? Private often means more secure. I don’t need a public IP for each node. Sounds good.

When you enable Private cluster you have to enter a CIDR IP class for you masters and your cluster will automatically become VPC-native. You can also enter IP numbering for containters and services of your choice.

It’s a very good idea to limit an access to the master for authorized networks only:

I also decided to install kubernetes dashboard and enable HTTP load balancing.

After a minute or two the cluster is ready and you can connect to it using kubectl. Well, but before this, you have to get cluster configuration and create kubectl context with gcloud:

gcloud container clusters get-credentials test-cluster --zone us-central1-a --project new-project

Ok, let’s check if it works:

kubectl run hello-world  --image=gcr.io/google-samples/node-hello:1.0 --port=8080
deployment.apps "hello-world" created

Looks good, let’s check if it works:

kubectl get deployments
NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-world   1         1         1            1           54s
kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
hello-world-78c5c876d4-ltdr5   1/1       Running   0          1m

Let’s try to expose this to the Internet:

kubectl expose deployment hello-world --port=8080 \
--type=LoadBalancer
service "hello-world" exposed
kubectl get svc
NAME          TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)          AGE
hello-world   LoadBalancer   10.0.13.194   35.225.14.164   8080:30321/TCP   44s
kubernetes    ClusterIP      10.0.0.1      <none>          443/TCP          19m

So far, so good. Let’s check it in the browser:

Great, we got exactly what we expected.

Lest’s try something more complicated:

kubectl run mariadb --image mariadb \
  --env="MYSQL_ROOT_PASSWORD=password"
deployment.apps "mariadb" created
kubectl get pods
NAME                       READY     STATUS             RESTARTS   AGE
mariadb-785d6974b7-5q8kk   0/1       ImagePullBackOff   0          1m

Wat? Why it can’t download the image?

Hmm.. let’s think.. we decided to have private cluster, do you remember? It’s time to RTFM. (I probably should do this before I started, but who does this?)

In a private cluster, the Docker runtime can pull container images from Google’s Container Registry. It cannot pull images from any other registry on the internet. This is because the nodes in a private cluster do not have external IP addresses, so they cannot communicate with sites outside of Google.

Ooooookey. So private means really private. Without access to the internet. This is secure for sure, but it’s not what I wanted. But maybe we can NAT the traffic through another instance with Internet access?

Unfortunately there is no managed entity like NAT Gateway in AWS, but we can do this manually via the instance.

Let’s create one:

gcloud compute instances create nat-gateway --network default \
  --can-ip-forward --zone us-central1-a --image-family=centos-7 \
  --image-project=centos-cloud  --boot-disk-size 10G \
  --machine-type f1-micro

Please note — the option --can-ip-forward is essential and can’t be changed later. Let’s ssh to this machine and configure NAT:

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -f
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
firewall-cmd --runtime-to-permanent

The last thing we have to configure is routing inside VPC. All default routes have priority 1000 so we have to use a smaller number to make the new route more important:

gcloud compute routes create nat-internet-route --network default \
  --destination-range 0.0.0.0/0 \
  --next-hop-instance nat-gateway \
  --next-hop-instance-zone us-central1-a \
  --tags no-ip --priority 900

Note the --tag no-ip option. This means this route will match the traffic from all instances with this network tag. Let’s tag our kubernetes node:

gcloud compute instances add-tags \
  gke-test-cluster-default-pool-11723ed6-7lh1 \
  --tags no-ip --zone us-central1-a

After a while:

kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
mariadb-67667f4b8c-lcss5   1/1       Running   0          7s

Of course this solution is temporary — it works until the node instance will crash or shutdown. The permanent solution is to create new instance template with this particular network tag no-ip and replace existing node pool with the new one based on this template. I’ll leave this as exercise for you ;).

Summary

It’s a very good idea to have kubernetes nodes in private network without direct access from/to the Internet. It’s more secure and proof for any human error while setting the firewall rules. However, to have it work with external Docker registry (for example Docker Hub) requires some additional work. Please note — the NAT instance we created is the SPOF. On the other hand we need it only when accessing resources outside Google. Another solution is to host all Docker images in the Google Image Registry which would make our cluster even more secure.