Canarytokens' new member: AWS API key Canarytoken

This is the fourth post in a series highlighting bits from our recent BlackHat USA 2017 talk. An index of all the posts in the series is here.

Introduction

In this blog post, we will introduce you to the newest member of our Canarytoken’s family, the Amazon Web Services API key token. This new Canarytoken allows you to sprinkle AWS API keys around and then notifies you when they are used. (If you stick around to the end, we will also share some of the details behind how we built it).

Background

Amazon Web Services offers a massive range of services that are easily integratable with each other. This encourages companies to build entire products and product pipelines using the AWS suite. In order to automate and manipulate AWS services using their API, we are given access keys which can be restricted by AWS policies. Access keys are defined on a per user basis which means there are a few moving parts in order to lock down an AWS account securely.

Take it for a spin - using an AWS API key Canarytoken

Using the AWS API key Canarytoken is as simple as can be. Simply make use of the free token server at http://canarytokens.org or use the private Canarytoken server built into your Canary console. Select the ‘AWS Keys’ token from the drop down list.



Enter an email and a token reminder (Remember: The email address is the one we will notify when the token is tripped, and the reminder will be attached to the alert. Choose a unique reminder, nothing sucks more than knowing a token is tripped, but being unsure where you left it). Then click on “Create my Canarytoken”.



You will notice that we arrange your credentials in the same way as the AWS console usually does, so you can get straight down to using (or testing) them. So lets get to testing. Click “Download your AWS Creds” and save the file somewhere you will find it.

For our tests, we are going to use the AWS Commandline tool (if you don’t have it yet, head over to http://docs.aws.amazon.com/cli/latest/userguide/installing.html). Below is a simple bash script that will leverage the AWS command line tool to create a new user named TestMePlease using your new-almost-authentic AWS API keys.

Simply go to your command line, navigate to the same location as the script and type, ./test_aws_creds.sh <access_key_id> <secret_access_key> . If all went to plan, you should be receiving an alert notifying you that your AWS API key Canarytoken was used.

NB: Due to the way these alerts are handled (by Amazon) it can sometimes take up to 20 minutes for the alert to come through.

Waiting...waiting...waiting (0-20mins later). Ah we got it!


Check...it...out! This is what your AWS API key Canarytoken alert will look like, delivered by email. The email will contain some useful details such as User Agent, Source IP and a reminder of where you may have placed this Canarytoken (we always assumed you not going to use only one! Why would you? They are free!!).

The simple plan then should be: Create a bunch of fake keys. Keep one on the CEO’s laptop. (He will never use it, but the person who compromises him will). Keep one on your webserver (again, no reason for it to be used, except by the guy who pops a shell on that box, etc)

Under the hood - steps to creating an AWS API key Canarytoken

The AWS API key Canarytoken makes use of a few AWS services to ensure that the Canarytoken is an actual AWS API key - indistinguishable from a real working AWS API key. This is important because we want to encourage attackers to have to use the key to find out how juicy it actually is - or isn’t. We also want this to be dead simple to use. Enter your details and click a button. If you want to see how the sausage is made, read on:


Creation - And on the 5th day…


The first service necessary for creating these AWS API key Canarytokens, is an AWS Lambda that is triggered by an AWS API Gateway event. Let’s follow the diagram’s flow. Once you click the ‘Create my Canarytoken’ button, a GET request is sent to the AWS API Gateway. This request contains query parameters for the domain (of the Canarytokens server), the username (if we want to specifiy one, otherwise a random one is generated) and the actual Canarytoken that will be linked to the created AWS API key. This is where the free version and commercial versions diverge slightly.

Our free version of Canarytokens (canarytokens.org), does not allow you to specify your own username for the AWS API key Canarytoken. The domain of the Canarytoken server is used in conjunction with the Canarytoken to create the AWS user on the account. (This is still completely useful, because the only way an attacker is able to obtain the username tied to the token, is to make an API call, and this call itself will trigger the alert). Our private Canary consoles enjoy a slightly different implementation. This uses an AWS Dynamo Database that links the users to their tokens and allowing clients the opportunity to specify what the user name for your AWS user should be. 

If the AWS API Gateway determines that sufficient information is included in the request, it triggers the lambda responsible for creating the AWS API key Canarytoken. This lambda creates a new user with no privileges on the AWS account, generates AWS API keys for that user and responds to the request with a secret access key and an access key id.


We should note that the newly created user has no permissions (to anything), so anyone with this AWS API key can’t do anything of importance. (Even if they did, its a user on our infrastructure, not yours!). Of course, before the attacker is able to find out how impotent her key is, she first has to use it and this is when we catch them out (detection time!).

Detection - I see you! 

Now that the AWS API key has been created and returned to the user, lets complete the loop and figure out when these AWS API keys are being used. The first service in our detection process, spoken about in our previous posts, is CloudTrail. CloudTrail is super useful when monitoring anything on an AWS account because it logs all important (not all) API calls recording the username, the keys used, the methods called, the user-agent information and a whole lot more. 

We configure CloudTrail to send its logs to another AWS logging service known as CloudWatch. This service allows subscriptions and filtering rules to be applied. This means that if a condition in the logs from CloudTrail is met in the CloudWatch service, it will trigger whichever service you configure it to - in our case another AWS Lambda function. In pure AWS terms, we have created a subscription filter which will send logs that match the given filter to our chosen lambda.

For the AWS API key Canarytoken, we use a subscription filter such as

  • "FilterPattern": "{$.userIdentity.type = IAMUser}"

This filter will check the incoming logs from CloudTrail and only send logs that contain the user identity as an IAM User - this is different from using root credentials as the user is then ‘root’.

Alert - Danger Will Robinson, danger!

All thats left now is for us to generate our Alert. We employ an AWS Lambda (again) to help us with this. This lambda receives the full log of the attempted AWS API call and bundles it into a custom HTTP Request that trips the Canarytoken. Our Canarytoken Server receives the request with all this information and relays the alert to you with all the information formatted neatly.

Summary - TLDR;

Amazon Web Services is a massive collection of easily integratable services which enables companies of all sizes to build entire products and services with relative ease. This makes AWS API keys an attractive target for many attackers.

The AWS API key Canarytoken allows the creation of real AWS API keys which can be strewn around your environment. An attacker using these credentials will trigger an alert informing you of his presence (and other useful meta information).. It’s quick, simple, reliable and a high quality indicator of badness.

Farseeing: a look at BeyondCorp

This is the third post in a series highlighting bits from our recent BlackHat USA 2017 talk. An index of all the posts in the series is here.


Introduction

In our BlackHat talk, "Fighting the Previous War", we showed how attacks against cloud services and cloud-native companies are still in their nascent stages of evolution. The number of known attacks against AWS is small, which is at odds with the huge number (and complexity) of services available. It's not a deep insight to argue that the number of classes of cloud specific attacks will rise.

However, the "previous war" doesn't just refer to cloud stuff. While our talk primarily dealt with cloud services, we also spent some time on another recent development, Google's BeyondCorp. In the end, the results weren't exciting enough to include fully in the talk and so we cut slides from the presentation, but the original slides are in the PDF linked above.

In this post we'll provide our view on what BeyondCorp-like infrastructure means for attackers, and how it'll affect their approaches.

What is BeyondCorp?

We start with a quick overview of BeyondCorp that strips out less important details (Google has a bunch of excellent BeyondCorp resources if you've never encountered it before.)

In an ossified corporate network, devices inside the perimeter are more trusted than devices outside the perimeter (e.g. they can access internal services which are not available to the public Internet). In addition, devices trying to access those service aren't subject to checks on the device (such as whether the device is known, or is fully patched).

In the aftermath of the 2009 Aurora attacks on Google, where attackers had access to internal systems once the boundary perimeter was breached, Google decided to implement a type of Zero Trust network architecture. The essence of the new architecture was that no trust was placed in the location of a client regardless of whether the client was located inside a Google campus or sitting at a Starbucks wifi. They called it BeyondCorp.

Under BeyondCorp, all devices are registered with Google beforehand and all access to services is brokered through a single Access Proxy called ÜberProxy.

This means that all Google's corporate applications can be accessed from any Internet-connected network, provided the device is known to Google and the user has the correct credentials (including MFA, if enabled.)

Let's walk through a quick example. Juliette is a Google engineer sitting in a StarBucks leaching their Wifi, and wants to review a bug report on her laptop. From their documentation, it works something like this (we're glossing over a bunch of details):
  1. Juliette's laptop has a client certificate previously issued to her machine.
  2. She opens https://tickets.corp.google.com in her browser.
  3. The DNS response is a CNAME pointing to uberproxy.l.google.com (this is the Access Proxy). The hostname identifies the application.
  4. Her browser connects using HTTPS to uberproxy.l.google.com, and provides its client certificate. This identifies her device.
  5. She's prompted for credentials if needed (there's an SSO subsystem to handle this). This identifies her user.
  6. The proxy passes the application name, device identifier (taken from the client certificate), and credentials to the Access Control Engine (ACE).
  7. The ACE performs an authorization check to see whether the user is allowed to access the requested application from that device.
  8. The ACE has access to device inventory systems, and so can reason about device trust indicators such as:
    1. a device's patch level
    2. its trusted boot status
    3. when it was last scanned for security issues
    4. whether the user has logged in from this device previously
  9. If the ACE passes all checks, the access proxy allows the request to pass to the corporate application, otherwise the request fails.
Google's architecture diagrams include more components than we've mentioned above (and the architecture changed between their first and most recent papers on BeyondCorp). But the essence is a proxy that can reason about device status and user trust. Note that it's determining whether a user may access a given application, not what they do within those applications.

One particularly interesting aspect of BeyondCorp is how Google supports a bunch of protocols (including RDP and SSH) through the same proxy, but we won't look at that today. (Another interesting aspect is that Google managed to migrate their network architecture without interruption and is, perhaps, the biggest takeaway from their series of papers. It's an amazingly well planned migration.)

This sucks! (For attackers)

For ne'er-do-wells, this model changes how they go about their business. 

Firstly, tying authorisation decisions to devices has a big limiting effect on credential phishing. A set of credentials is useless to an external attacker if the authorisation decision includes an assertion that the device has previously been used by this user. Impersonation attacks like this become much more personal, as they require device access in addition to credentials.

Secondly, even if a beachhead is established on an employee's machine, there's no flat network to laterally move across. All the attacker can see are the applications for which the victim account had been granted access. So application-level attacks become paramount in order to laterally move across accounts (and then services).

Thirdly, access is fleeting. The BeyondCorp model actively incorporates updated threat information, so that (for example), particular browser versions can be banned en masse if 0days are known to be floating around. 

Fourthly, persistence on end user devices is much harder. Google use verified boot on some of their devices, and BeyondCorp can take this into account. On verified boot devices, persistence is unlikely to take the form of BIOS or OS-level functionality (these are costly attacks with step changes across the fleet after discovery, making them poor candidates). Instead, higher level client-side attacks seem more likely.

Fifthly, in addition to application attacks, bugs in the Access Control Engine or mistakes in the policies come into play, but these must be attacked blind as there is no local version to deploy or examine.

Lastly, targeting becomes really important. It's not enough to spam random @target.com addresses with dancingpigs.exe, and focus once inside the network. There is no "inside the network", at best you access someone's laptop, and can hit the same BeyondCorp apps as your victim.

A quick look at targeting

The lack of a perimeter is the defining characteristic of BeyondCorp, but that means anyone outside Google has a similar view to anyone inside Google, at least for the initial bits needed to bootstrap a connection.

We know all services are accessed through the ÜberProxy. In addition, every application gets a unique CNAME (in a few domains we've seen, like corp.google.com, and googleplex.com).

DNS enumeration is a well-mapped and frequently-trod path, and effective at discovering corporate BeyondCorp applications. Pick a DNS enumeration tool (like subrute), run it across the corp.google.com subdomain, and get 765 hostnames. Each maps to a Google Corporate application. Here's a snippet from the output:
  • [...]
  • pitch.corp.google.com
  • pivot.corp.google.com
  • placer.corp.google.com
  • plan.corp.google.com
  • platform.corp.google.com
  • platinum.corp.google.com
  • plato.corp.google.com
  • pleiades.corp.google.com
  • plumeria.corp.google.com
  • [...]
But DNS isn't the only place to identify BeyondCorp sites. As is the fashion these days, Google is quite particular about publishing new TLS certificates in the Certificate Transparency logs. These include a bunch of hostnames in  corp.google.com and googleplex.com. From these more BeyondCorp applications were discovered.

Lastly, we scraped the websites of all the hostnames found to that point and found additional hostnames referenced in some of the pages and redirects. For fun, we piped into PhantomJS and screencapped all the sites for quick review.

Results? We don't need no stinking results!


The end result of this little project was a few thousand screencaps of login screens:

Quite a few of these
Error showing my device isn't
allowed access to this service
Occasional straight 403


So, so many of these
Results were not exciting. The only site that was open to the Internet was a Cafe booking site on one of Google's campuses.

However, a few weeks ago a high school student posted the story of his bug bounty which appeared to involve an ÜberProxy misconfiguration. The BeyondCorp model explicitly centralises security and funnels traffic through proxy chokepoints to ease authN and authZ decisions. Like any centralisation, it brings savings but there is also the risk of a single issue affecting all applications behind the proxy. The takeaway is that mistakes can (and will) happen. 


So where does this leave attackers?

By no means is this the death of remote attacks, but it shifts focus from basic phishing attacks and will force attackers into more sophisticated plays. These will include more narrow targeting (of the BeyondCorp infrastructure in particular, or of specific endusers with the required application access), and change how persistence on endpoints is achieved. Application persistence increases in importance, as endpoint access becomes more fleeting.

With all this said, it's unlikely an attacker will encounter a BeyondCorp environment in the near future, unless they're targeting Google. There are a handful of commercial solutions which claim BeyondCorp-like functionality, but none rise to the same thoroughness of Google's approach. For now, these BeyondCorp attack patterns remain untested.