The AWS API Key Canarytoken (paid and free) is a great way to detect attackers who have compromised your infrastructure. The full details are in a previous blogpost, but in short:
- You go to https://canarytokens.org and generate a set of valid AWS API credentials;
- Simply leave those in ~/.aws/config on a machine that’s important to you
- Done!
If that machine is ever breached, the sort of attackers who keep you up at night will look for AWS API credentials, and they will try them.
And when they do, we let you know that you’ve been breached.
When you receive an email/SMS/Slack message letting you know that the AWS API key that you left only on BuildServer-7 in Server-room #12 just got used to login to AWS, you know you have a problem.
The underlying Canarytoken infrastructure relies on AWS APIs logging their own execution to CloudTrail. This lets us identify: which IP made the call; which API was executed (including both the service name and function); plus other details about the client executing the call.
The effectiveness of the AWS API Key Canarytoken shouldn’t be underestimated. Attackers have to try them; they could be the keys to the victim’s kingdom. If they don’t try the keys, they might be missing a golden opportunity. AWS API Keys are about the juiciest bait you can dangle in front of adversaries.
For defenders, the keys we supply are simple and entirely safe. They’re not tied to anything owned by the defender (the infrastructure sits completely at Thinkst), the keys have no permissions so they can’t be used to do anything, and no agents or software needs to be installed. You simply drop a text file, and wait to be notified if anything happens.
Between Canarytokens.org and our commercial Canarytokens offering, thousands and thousands of machines worldwide have AWS API Canarytokens lying in wait for attackers.
Can AWS Canarytokens be detected?
This introduces a new goal for the super stealthy attacker. If they find an AWS API key on a server, can they tell if it’s a Canarytoken (or if it is a legitimate key?). Our view is that this only matters if an attacker can determine this without triggering the token. In other words, the Canarytoken fails if attackers can come across API credentials, and perform some test that returns whether or not the credentials are Canarytokens but defenders are never notified of the test.
The key (heh) to making this happen as an attacker, would be to find AWS APIs that don’t log their execution, and also reveal information about the calling API Key to the caller.
For our purposes, there are four classes of error responses from AWS APIs:
- Logs to CloudTrail, reveals no information about the API key
- Logs to CloudTrail, does reveal account details from the API key
- No CloudTrail logs, reveals no information about the API key
- No CloudTrail logs, does reveal account details from the API key
As defenders, 4 is the worst case. If an API fulfills number 4, then Canarytokens can be detected. 3 is not great either, but softens the impact to “attackers can determine where credentials are valid, but not whether they’re Canarytokens”.
The next bit may surprise those who’ve never worked with it: the AWS API is a mishmash of response codes, error code, error strings, exception names, and data formatting. It is anything but consistent.
In the past, it’s been the case that a tiny subset of API errors did fulfill the constraints for class 4. The folks at Rhino Security blogged about this (RIP Spencer). While AWS (very slowly) does seem to react to reports on APIs which don’t show up in CloudTrail, that’s not sufficient. As it stands, we’re not aware of class 4 error responses currently in the AWS API, but at the rate at which new APIs are added, that’s little solace for thinking about the future.
There are certainly class 3 error responses in the current AWS API if you go looking hard enough.
In summary, the AWS API has a small number of endpoints which don’t log their own usage to CloudTrail, creating a blindspot for defenders. What do we do in those instances?
IAM Credential Reports
Scott Piper pointed out that Amazon actually does provide a backup mechanism for identifying credential usage. The IAM service lets you generate and download a report for all your credentials. This is a CSV file where each row belongs to an IAM user, and some of the columns identify when an access key was used, and on which AWS service it was used.
With the credential report, there are no longer class 3 or 4 error responses in a practical sense; we can also check the credential report to see if an API key was used, so no longer need to just rely on CloudTrail.
The report has three main drawbacks.
- The report can only be generated once every four hours, so at worst there’s a four hour delay between credentials being used, and you seeing the notifications.
- The fidelity of the reports is low; you only can tell the last time the key was used and on which service (e.g. ec2, iam, sts, etc). You can’t tell which function was called.
- You don’t get any client information, including IP addresses or client versions.
However in spite of these drawbacks, it’s a huge step up in terms of reliability of this token type. Attackers don’t have places to sneakily test their API keys. This safety net means that every key usage will be detected and alerted on, albeit sometimes with lower fidelity artifacts.
Even if you didn’t know the attackers IP Address, would it be worth knowing that the key that you placed on \\code-sign-22 was used to login to AWS this morning? Absolutely!
In fact, if a key is only used on what was previously a class 3 or 4 API, that’s even more of a signal since it implies the attacker is actively trying to avoid detection.
So we built an AWS safety net, to help us catch attackers who actively try to avoid detection.
Building an AWS Safety Net
The architecture is very straight forward. An AWS Lambda function fires on a periodic schedule. It pulls in the credential report from IAM, iterates over all the rows, and determines whether the last recorded use of the key happened more recently than the CloudTrail logs show. If so, the Safety Net kicks in, and sends out the alert:
Browsing to the history page for the incident, you see an annotation telling you that the Safety Net picked up this usage:
The Safety Net was deployed to our commercial customers a little while back thanks to Jason, and we’ve recently rolled it out to the free users on Canarytokens.org too.
Wrapping Up
AWS API logging previously left a small but significant gap that potentially gave attackers a way to use Canarytoken API keys without triggering alarms. With the deployment of the Safety Net, this gap has been plugged.