Skip to main content

Get NodeAuthorization working in Kubernetes with acs-engine

Node Authorization in k8s

I'm starting to get into the container world and I'm loving it. Recently we helped a client build out and deploy a micro-services application in Kubernetes. We created the cluster in Azure using the open source project acs-engine. After we got the cluster set up, our client asked for some updates to the cluster for security reasons. One of those updates was to enable Node Authorization.

What is Node Authorization?

Node Authorization locks down each Node in the cluster to only be able to do actions on itself. If this is not turned on, its possible for a malicious pod to take actions on any other node, including reading secrets, deleting pods, etc. There is an excellent post by Antoine Cotten that explains this very well ( as well as RBAC, which is a different subject altogether).

How do I set it up?

Based on the current documentation, it looks like setting up Node Authorization should be easy. Basically follow these steps
  1. Turn on TLS Boot Strapping
  2. Edit the api-server yaml file
  3. Sit back and relax knowing your cluster is more secure
The trouble is that #1 takes you down a very deep rabbit hole of links starting here. The rest of this post will walk you through how I got it set up.

Turning on TLS Bootstrapping

The first step in enabling TLS bootstrapping is to create a token csv file on the master node and update the apiserver yaml file to use it.

Create Token file

  1. SSH into the master node
  2. change directory to /etc/kubernetes/
  3. run the following command to create a token string 
    1. head -c 16 /dev/urandom | od -An -t x | tr -d ' '
  4. create a bootstrap.csv file
    1. touch bootstraptoken.csv
  5. edit the file
    1. vi bootstraptoken.csv
  6. Add the following text, substituting in your token created above
    1. [token],kubelet-bootstrap,10001,"system:bootstrappers"
    2. example: c0e89sdfasdfasdfff3b8718ab7868cc989,kubelet-bootstrap,10001,"system:bootstrappers"

Edit API Server

After the CSV file is created, we need to update the api server configuration to use it. 
  1. Make sure you are still SSH'ed into the master
  2. edit the api server configuration at :/etc/kubernetes/manifests/kube-apiserver.yaml
    1. vi /etc/kubernetes/manifests/kube-apiserver.yaml
  3. Add "--token-auth-file=/etc/kubernetes/bootstraptoken.csv" to the commands
Great! Now your master node is set up to allow TLS bootstrapping. Next up is to get your agents set up.


Configure Agent Node

Note: the following instructions are to configure a single agent. You will need to do this for each agent in your cluster.

  1. SSH to the Agent node. 
    • Typically this is done by SSH'ing to the master, uploading your private key to the master and then SSH'ing into the master.

Edit kubeConfig file

We need to edit the create a kubeconfig file that uses the token string created above.

  1. Change directory to /var/lib/kublet
    1. cd /var/lib/kubelet
  2. Create a bootstrap.kubeconfig file
    1. touch bootstrap.kubeconfig
  3. Edit the bootstrap.kubeconfig file with the following text, substituting in the IP address of the master node and the token string created above
    1. apiVersion: v1
      kind: Config
      clusters:
      - name: localcluster
        cluster:
          certificate-authority: /etc/kubernetes/certs/ca.crt
          server: https://MASTERNODEIP:443
      users:
      - name: kubelet-bootstrap
        user:
          token: TOKENSTRING
      contexts:
      - context:
          cluster: localcluster
          user: kubelet-bootstrap
        name: localclustercontext
      current-context: localclustercontext

Edit Kublet

The next step is to edit the Kublet configuration to point to the bootstrap.kubeconfig file created directly above
  1. Change Directory to /etc/systemd/system
  2. Edit kubelet.service
    1. Add the following to the startup section, making sure to include the forward slash

Restart Kubelet

After those edits, let's restart the Kubelet to make sure that it picks up the latest changes. Run the following commands: 

systemctl daemon-reload
systemctl stop kubelet
docker kill $(docker ps -q)
systemctl start kubelet

Validate it's working

Now that the master and agents are updated, lets make sure its working

  1. run "kubectl get nodes"
    1. the node you just updated should say "NotReady". This is because we have not approved the CSR for the node. Lets do that now
  2. run "kubectl get csr"
    1. this will show the CSR's that have been submitted. You should see one in a pending state for the node was just set up
  3. run "kubectl describe certificate to review the request. The Common Name should be "System:nodes:
  4. Run "kubectl get nodes" again. This time the node should be back to "Ready" state

Edit API Server (again)

Now TLS Bootstrapping is set up, but NodeAuthorization isn't enabled yet. In order to do that, we need to go back to the master and edit the api server.

Notes: 
  • Don't do these steps as part of editing the masters above as your cluster will become unstable since the API server will expect the agents to be configured with TLS Bootstrapping.
  • While the documentation on NodeAuthorization doesn't mention needing RBAC enabled, i was unable to schedule pods on nodes until I turned it on. There is most likely some reliance on it, or an additional configuration I missed. However enabling RBAC is probably a good idea from a security standpoint anyway.

  1. SSH Into the Master
  2. Change directory to /etc/kubernetes/manifests
  3. edit kube-apiserver.yaml and add the following commands:
    1. "--authorization-mode=RBAC,Node"
    2. Add "NodeRestriction" to the existing admission-control command
    3. example:
  4. Restart all docker images on the master
    1. docker kill $(docker ps -q)

Summary

In summary, the steps above aren't daunting and but aren't easy to figure out from the docs. Adding this capability into acs-engine wouldn't be too difficult, even if you just customized it for your own cluster.

A big thanks to Lachie Evenson for helping me get this running. He's got some great videos around Kubernetes that I've found very useful. 


 Hopefully this saves someone an hour (or day) or two. Enjoy!


    Comments

    Popular posts from this blog

    MSDEPLOYAGENTSERVICE 401 unauthorized–Resolution

    We recently migrated a production environment for a client to new Servers. I had previously been using MSDeploy to deploy the websites/services to the servers so I figured all I had to do was install MSDeploy, point Update my deploy scripts to point to the new servers, and deploy! I was using MSDeploy 2 on the previous servers so I figured it would work on the new ones. Unfortunately it didn’t turn out to be that easy.
    When I ran the updated scripts I got the following error:
    Fatal: Request to remote agent URL 'http://myserver/MSDEPLOYAGENTSERVICE' failed.
    Fatal: The remote server returned an error: (401) Unauthorized. Fatal count: 1
    I was using an admin account and I could hit that URL above in a browser so I knew it wasn’t an authorization issue.
    Here are the things I tried that DIDN’T work:
    Uninstall/Reinstall MSDeploy 2Install MSDeploy 3Create the fake user group on the server per these instructions:http://www.iis.net/learn/publish/troubleshooting-web-deploy/web-deploy-er…

    Build/Deploy Windows service with TFS

    Building and deploying a web service or website via TFS is pretty straight forward, but automatically deploying a windows service or console application takes a b it of setup. I’ve outlined the steps below to get your service deployed. Step 1: Set up Automated Build and MSDeploy on the Target server. If you are doing any sort of automated build/deploy I recommend using MSDeploy. You can follow steps 1-3 from a previous post here.Step 2: Edit the .csproj fileThe project file for the app you are trying to deploy needs to be edited so that TFS will create a directory for the output of the project. Otherwise the project will be built, but the assemblies will be placed in the code drop location with no organization to it.To edit the file, go to the open file dialog and locate the csproj file. Select it but don’t click “Open. Instead click on the dropdown arrow next to the Open button, select “Open with”, and choose the XML (Text) editor as shown:Once the file is open, add the following “Pr…