“What will an attacker do if they find an AWS API key on your server?” (We are pretty convinced they will try to use it, and when they do, you get a reliable message that badness is going on).
Last month we asked:
“What will an attacker do if they find a large MySQLDump file on your machine?” (We think there’s a good chance they will load it into a temp MySQL db, and when they do, you get a reliable message that badness is going on).
This month, a similar question comes to the container world:
“What will an attacker do if they find a good looking kubeconfig file on one of your servers?”
If the answer is: “They will try to use it to access your kubernetes cluster”, then again, you will receive a high-fidelity alert that badness is happening.
This quick post presents our shiny new Kubeconfig Token (which emulates a kubeconfig file, the configuration text file that ordinarily contains credentials to interact with a Kubernetes cluster).
A Canarytoken refresher
Canarytokens are a great way to tripwire important servers and locations. With just a few clicks you get to drop legitimate looking resources on your network that alert you when they’re used. Chosen correctly, the Canarytoken is impossible for an attacker to resist while also guaranteeing an alert when it is used or accessed. This sort of detection tactic is super powerful because it becomes technology agnostic while exploiting the attackers objectives i.e. it doesn’t matter if she broke into your network/cloud via a phishing attack or via a mega complex supply chain attack. What matters is that she is there, and has objectives. If she finds a key to what could be your AWS presence, can she avoid using it? If she finds a doorway to one of your Kubernetes clusters, can she resist popping in?
It’s not just about attacker curiosity. Most sophisticated attacks are less one massive blow, and more like “death by a 1000 cuts”. With Canary (and Canarytokens) each of those cuts is an opportunity to reliably detect the compromise early on.
We know that Canarytokens work because we’ve seen them used to catch attackers, pen-testers and nation-state adversaries around the globe. We also have great attacker feedback that quickly gets to the core:
Attackers are forced to slow down and are forced to work harder when they don’t know what they can trust. As Paul McMillan points out, more tokens widely deployed actually slows down attackers everywhere since they increasingly can trust fewer things.
Like all Canarytokens, using it is simple and straightforward:
Surf to canarytokens.org and select the kubeconfig token
Enter your email address (to receive the alert) and enter a memo to remind you where you are dropping this token
We give you a working config file that you leave on the server and forget about.
That’s it! Your work is done here!
If weeks or months from now, an attacker does manage to SolarWinds you, they’re going to find that config, and will probably use it to extend their compromise. And you will get a reliable notification that something isn’t kosher on \\CodeSignServer-03
The Kubeconfig Canarytoken can be placed on any server to make it seem like a Kubernetes node, or even on your engineers’ machines to make them seem like systems that regularly interact with your production Kubernetes cluster. Kubeconfigs are also used in CI/CD pipelines to authenticate to a cluster and utilize it for running jobs, so it can also be placed on a server used as a node in the pipeline.
Background: The kubeconfig file
A kubeconfig is a YAML formatted file that contains all materials needed to authenticate to the control plane of a Kubernetes cluster. This includes the name of the cluster, API Server endpoint and the user information.
This file is typically used with kubectl, which is a command-line utility used to run commands against a Kubernetes cluster. The kubeconfig file can be used with kubectl by:
Specifying its path as the value of the –kubeconfig flag when running kubectl
Specifying its path as the value of the KUBECONFIG environment variable
Placing its contents at $HOME/.kube/config (this is the most common approach)
Our Kubeconfig Token is a kubeconfig file generated by us that can be used just like any other kubeconfig file — it contains the API server endpoint that kubectl should connect to, as well as the credentials needed to access the API server. These credentials are unique to your Canarytoken, and are how the token works. Using the kubeconfig file will simply return permission errors for an attacker, and result in a neat, timely alert for you!
Background: Why Kubernetes?
Kubernetes is a beast. It’s the most popular container orchestrator and has continued to be adopted widely at a steady pace, even in odd places. The U.S. Army runs Kubernetes on their F-16 fighter jets?!
But, because Kubernetes is also a complex, often misunderstood and misconfigured beast, it is easy to get things wrong when setting up a cluster. Predictably, malicious actors now routinely search for Kubernetes-specific environment-markers in their recon phase post-compromise of internet exposed services.
Palo Altos Unit42 have documented recent attacks against containerized workloads running on Kubernetes clusters in the cloud where attackers either hunt specifically for the kubeconfig file or employ malware that searches for files having the same structure: