INTRODUCTION
Continually refining our security operations is part and parcel of what we do at Thinkst Canary to stay current with attacker behaviours. We’ve previously written about how we think about product security (where we referenced earlier pieces on custom nginx allow-listing, sandboxing, or our fleet-wide auditd monitoring).
Recently we examined our exposure to API key leakage, and the results were unexpected.
THIRD PARTY API KEYs
Like most companies, we use a handful of third-party providers for ancillary services. And, like most providers, they expose an API and give us an API key. A short time back as part of an exercise in examining our internal controls relating to third-party API keys we asked:
- has an attacker grabbed this key?
- has she actually used this key ?
- what did she do with this key?
It turns out that even really popular service providers, by default, provide very crumby answers to these questions.
That’s quite a conclusion to reach.
To be clear, most providers expose their service logs (i.e. what did they do for you), but few expose API logs (what Thinkst’s API key did or attempted on the service). Consider a third-party transactional mailer service which sends emails on your behalf. An API key lets you send emails, but it also lets you query the API to recover previously sent emails (within a time window). With one provider there’s simply no way for us to determine whether our API key has ever been used to retrieve old emails; API logs aren’t available.
The tl;dr is that if we expect to have answers to these questions, we have to take care of creating those logs ourselves. But how?
OUR USE OF API KEYs
In our current model, we have hundreds of customer Consoles that hold an API key for mail and SMS providers. Both our mail and SMS providers restrict us to a single key, so we end up with all those consoles using a shared key.
This isn’t a train smash, but what if an attacker compromised a single customer console?
With enough privileges, they’d be able to grab those keys and start to query the provider APIs. (It would be great if we could have separate keys, or if we could ask useful questions of the providers). We can’t, so we built the “broker”.
THE BROKER
Conceptually it’s pretty straightforward. The Broker is a proxy which sits between our Consoles and third-party providers. The actual API keys are stored on the Broker, and not the Consoles. So a breached Console cannot reveal valid API keys (because they’re absent). Instead, we generate replacement keys, unique to each Console, as part of our configuration management process.
As a single go-binary, the broker has a pretty small attack-surface, and logs all of its actions to our ELK stack.
This position on the hot path between API consumer and provider lets us add a bunch of coolness:
- We create individual keys for each of our Consoles, even if the provider only gives us one. That means a breach on any Console doesn’t yield access to the third-party API key, and activity can be traced back to the individual Console;
- We log all interactions. This gives us our audit trail which the API providers are unable or unwilling to expose;
- We can cycle keys on a penny (and have added this into our configuration process);
- (Down the line we can even do things like block certain API calls to certain keys).
This is conceptually similar to Diogo Mónica and Nate McCauleys crypto-anchors (which is a 2014 talk worth watching)..
LIMITATIONS
An API broker like this isn’t for everyone. We now have an extra service to maintain and there’s a single service holding multiple API keys (but we are pretty happy with the trade-off).
In terms of scaling, while a proxy is a chokepoint, Canaries don’t generate loads of alerts and we haven’t seen bottlenecks (should that situation arise, load balanced Brokers are possible but not worth the added complexity right now).
We considered using one of the big API gateways, but decided against it. While they seem to focus on API’s going in the other direction they can be twisted to suit our needs. We just didn’t want a huge, relatively untrusted code-base fulfilling this role for us.
WHAT’S NEXT?
We’ve been running the Broker internally for a while and every day thousands of emails and SMSs are sent through the little broker that could. We’ll add a little more documentation and will throw it up on our Github account for others to use. If you are interested in playing with it, drop us a note @ThinkstCanary or at research@thinkst.com