Something fresh

This month we’re ready to release our first major Canary Console overhaul. We’ve obviously pushed updates to Canary and the Console weekly for almost 5 years but this is the first time we’ve dramatically reworked the Console.

Contrary to a bunch of other products, we don’t want to be your single  pane of glass, and work really hard to make sure that most customers never have to spend time in their Console at all. But our beefed up Console offers you a bunch of  fresh possibilities, and we figured we’d introduce some of them here.

What’s different?

The first thing that a new user should notice, is that it doesn’t feel that different to the old Console. It has a new coat of paint, and some things look slicker, but it feels like just a slight visual upgrade on the original Console.

This is completely by design, and belies a bunch of changes beneath the surface. It’s practically a trope that just as users become familiar with a product, the vendor drastically alters the user interface forcing users to re-learn flows which were previously easy. We hate this. Tools are supposed to make our lives easier, not periodically give us pop-quizzes.
We know that there’s a fine-line between keeping the product familiar, and introducing new features (or refreshing the look). Throughout this process we’ve tried to keep a clear view of this line. We’re really excited to show a few of the enhancements that are deployed to customers as of today.

From the screenshot above, you can see a few of these right off the bat. 


The new search-box at the top means that you never have to hunt for things again. When we first built Canary, customer Consoles had 5–10 birds in them (and no Canarytokens). Today, we have Consoles with hundreds of birds and tens of thousands of active Canarytokens. The search-box allows us to find things, even if we aren’t really sure what we are looking for. The search feature will let you find Bird, Incidents, Canarytokens and more without having to hunt, and you can search on pretty much any data tied to them.

A better graph-view

We’ve made a heap of improvements to our graph-view, which displays your alerts graphically rather than in table form. This is especially useful if you get a bunch of alerts; a quick click on the graph-view button will immediately clarify if it’s an attack sourced from a single or multiple locations in your network, and show you the birds involved. 

Birds of a Feather

The biggest improvement with the new Console is the ability to group birds into flocks. We spent heaps of time making this simple and intuitive so using it should feel pretty natural.

Once you’ve created a flock, any birds or tokens you previously had enrolled will be sitting in your “Default Flock”. These can be moved over to new flocks if you choose.

Of course the point (and joy) of having different flocks, is that you can treat them differently, so all of your flocks can have different settings, different users, and even different alerting rules.

User Management

Although we’ve supported the ability to add and remove users from your Console for a while, you now have much finer grained control of your users and their permissions. You can add users, restrict them to just a single flock, allow them to only deal with alerts, and delegate managing the actual birds to further users. The permissions model is simple: you can watch flocks, or manage them.


Canarytokens also gets a refresh in the new Console but it brings a bunch of utility that is deserving of its own post (next week). It’s especially useful to be able to place them in different groups. Shortly we’ll be releasing updates to the Console API alongside helper utilities to make it easier to deploy them by the dozen (or dozen dozen) inside your networks.

Audit Trail

If you’re an admin user, you now have access to the Audit Trail, which gives you detailed information on all activities performed on the Console. (You can also download a JSON dump of all activity if needed). The audit trail backend code has been in place for a while, so your audit trail is already populated with a bunch of your activity.

Support and our Knowledge Base

We try hard to make sure that Canary is easy to use, and where options need explanations, we usually build this into the app. However it’s still possible for users to lose their way. The Console now includes a constantly visible link to “help” that’s backed by a heavily populated Knowledge Base and a pretty decent search. You can still use the interface to send us a support request, and our helper elves will be super-quick to respond, but the KB should make things easier.

Email Changes

The new look also means that things like your weekly newsletter get a much-awaited visual upgrade but some changes are also functional.

Emails now include single-use buttons, to Acknowledge or Delete alerts (or to add them to your ignore list) which don’t require you to login. (This allows you to react quickly from your phone/mailbox. We really mean it when we say we don’t want to be your single pane of glass.)

Easy copy & paste

We’ve added a bunch of convenience functions to make sure that getting data out of the Console is quick and simple. Most data fields have a neatly hidden Copy button you can hit to grab data into your clipboard.


We already support SSO and you’ve always been able to make use of Duo / Google Authenticator / Authy for MFA. The new Console adds the awesomeness that is WebAuthn to our authentication defense lineup.

More to come

There’s  a bunch of other features that we can’t wait to share with you, and in the coming days will release more blogposts. For now, take it for a spin. It should be all Canary: Simple, and easy to use. Drop us a note with your thoughts!

3rd-party API-Key Leaks (and the Broker)


Continually refining our security operations is part and parcel of what we do at Thinkst Canary to stay current with attacker behaviours. We’ve previously written about how we think about product security (where we referenced earlier pieces on custom nginx allow-listing, sandboxing, or our fleet-wide auditd monitoring).

Recently we examined our exposure to API key leakage, and the results were unexpected.


Like most companies, we use a handful of third-party providers for ancillary services. And, like most providers, they expose an API and give us an API key. A short time back as part of an exercise in examining our internal controls relating to third-party API keys we asked:
  • has an attacker grabbed this key?
  • has she actually used this key ?
  • what did she do with this key?
It turns out that even really popular service providers, by default, provide very crumby answers to these questions. 

That’s quite a conclusion to reach.

To be clear, most providers expose their service logs (i.e. what did they do for you), but few expose API logs (what Thinkst’s API key did or attempted on the service). Consider a third-party transactional mailer service which sends emails on your behalf. An API key lets you send emails, but it also lets you query the API to recover previously sent emails (within a time window). With one provider there’s simply no way for us to determine whether our API key has ever been used to retrieve old emails; API logs aren’t available.

The tl;dr is that if we expect to have answers to these questions, we have to take care of creating those logs ourselves. But how?


In our current model, we have hundreds of customer Consoles that hold an API key for mail and SMS providers. Both our mail and SMS providers restrict us to a single key, so we end up with all those consoles using a shared key. 

This isn’t a train smash, but what if an attacker compromised a single customer console?

With enough privileges, they’d be able to grab those keys and start to query the provider APIs. (It would be great if we could have separate keys, or if we could ask useful questions of the providers). We can’t, so we built the “broker”.


Conceptually it’s pretty straightforward. The Broker is a proxy which sits between our Consoles and third-party providers. The actual API keys are stored on the Broker, and not the Consoles. So a breached Console cannot reveal valid API keys (because they’re absent). Instead, we generate replacement keys, unique to each Console, as part of our configuration management process.

As a single go-binary, the broker has a pretty small attack-surface, and logs all of its actions to our ELK stack.

This position on the hot path between API consumer and provider lets us add a bunch of coolness:
  • We create individual keys for each of our Consoles, even if the provider only gives us one. That means a breach on any Console doesn’t yield access to the third-party API key, and activity can be traced back to the individual Console;
  • We log all interactions. This gives us our audit trail which the API providers are unable or unwilling to expose;
  • We can cycle keys on a penny (and have added this into our configuration process);
  • (Down the line we can even do things like block certain API calls to certain keys).
This is conceptually similar to Diogo Mónica and Nate McCauleys crypto-anchors (which is a 2014 talk worth watching)..


An API broker like this isn’t for everyone. We now have an extra service to maintain and there’s a single service holding multiple API keys (but we are pretty happy with the trade-off).

In terms of scaling, while a proxy is a chokepoint, Canaries don’t generate loads of alerts and we haven’t seen bottlenecks (should that situation arise, load balanced Brokers are possible but not worth the added complexity right now).
We considered using one of the big API gateways, but decided against it. While they seem to focus on API’s going in the other direction they can be twisted to suit our needs. We just didn't want a huge, relatively untrusted code-base fulfilling this role for us.


We’ve been running the Broker internally for a while and every day thousands of emails and SMSs are sent through the little broker that could. We’ll add a little more documentation and will throw it up on our Github account for others to use. If you are interested in playing with it, drop us a note @ThinkstCanary or at