Home

We discussed this Scott Forstall clip internally and figured it was worth sharing since theres so much going on in just 5 minutes.

Bradley commented on how familiar it felt to how we roll and it’s worth digging in to this little more.

Quick Background

In 2007 Apple was not yet a trillion dollar company, but its star was definitely on the rise. Jobs was back, OSX was taking root and the iPod was game changer. (Their market cap was ~$174 Billion).

Demo Prep.

They are about to demo the iPhone in a few months and already you notice them sweating the presentation details. Should the CEO of AT&T be on stage? When do we read him in?

You can watch a zillion tech demos, and you will find demo’ers who look like they practiced on the plane ride in.. instead, this is thoughtful craft.

Getting AT&T excited

So they are flying to Vegas to show AT&T the phones.

Notice again, it’s not just “here’s an invite to our launch – you can have a speaking slot” where some big-shot CEO can waffle about the synergy between the two companies. They want him bought in..

They are going to demo to blow his socks off because that’s going to make his talk different too..

(On a much smaller scale) We’ve been using this technique for a while too. For almost every podcast we’ve advertised on, we first requested a private demo with the hosts. You can tell the difference in how they read our ad before and after seeing the demo. You can pay hundreds of thousands of dollars, but you can’t buy that difference in their voice when they’ve seen the win.

Leaving things to chance

Now.. the iPhone is clearly revolutionary… In another part of the interview (included in the full clip) Forstall mentions how at that point, it was obvious to them that the iPhone would win. He was using it, and knew it was the future, but with all that confidence they were still leaving nothing to chance.. they are going to demo it and Forstall is planning around it:

• Is there going to be good cell signal/service in the penthouse or will our demos look bad because of it?

Again. It’s such dedication to the craft. They have the Jesus-phone. You are talking to a seasoned exec. You could totally hand wave past the bad connectivity.

But.They.Don’t.

They want to make sure it pops out of their bag and works.

The option for using 4-Seasons Wi-Fi is great and what’s clear again is their dedication to removing the suck from the experience. Anyone would have understood “let me just join the Wi-Fi… -fiddle- these pesky things -fiddle-“ but they don’t. They are working on it before they leave Cupertino.

I also think it’s awesome that Forstall himself is sweating these details. He’s the head of engineering at a $100 billion dollar company. He isn’t sitting in the suite while IT makes it happen and this isn’t just fake egalitarian kayfabe. It’s a hardwired culture of biasing to action / just making it happen.

Relentlessly Resourceful

(It’s subtle, but) Notice also, that they didn’t land the solution because he played his Apple Card. He landed it by acting like an AT&T exec and using their weight. Anyone coulda done this (but very few ppl woulda).

Paul Graham has a famous term when talking about great startup founders. He calls them “relentlessly resourceful.”

A couple days ago I finally got being a good startup founder down to two words: relentlessly resourceful.

Till then the best I’d managed was to get the opposite quality down to one: hapless. Most dictionaries say hapless means unlucky. But the dictionaries are not doing a very good job. A team that outplays its opponents but loses because of a bad decision by the referee could be called unlucky, but not hapless. Hapless implies passivity. To be hapless is to be battered by circumstances—to let the world have its way with you, instead of having your way with the world.

Unfortunately there’s no antonym of hapless, which makes it difficult to tell founders what to aim for. “Don’t be hapless” is not much of rallying cry.

…

But finally I’ve figured out how to express this quality directly.

…

What would someone who was the opposite of hapless be like? They’d be relentlessly resourceful. Not merely relentless. That’s not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can’t simply plow through them, because you don’t know initially how hard they are; you don’t know whether you’re about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.

Be relentlessly resourceful.

What you see from the Forstall conversation (which seems obvious in retrospect) is that this isn’t just a requirement for startups. Relentlessly Resourceful is why Apple pulls off the iPod with a small team. It’s why they prototype the iPhone with a tiny team. It’s why they consistently manage to do what they do. Lots and lots of companies have more engineers than they do, and lots of companies have smart execs… but.. all the money can’t buy you smart execs / engineers who default to action and find their way around walls (instead of just pointing out that the walls exist). That bit is culture.

Relentlessly Resourceful can almost be written as “always be hacking”. We built Thinkst for this. We don’t just want the hacks to build our company, we built our company so we can do awesomer and awesomer hacks.

So far, it’s been great for us. It’s leet to see (when you look at Apple) that it scales 🙂

Given the importance of REST API endpoints for most networks and applications, we wanted a way to use (existing) Canarytokens, or Canaries to detect unauthorized access to a REST API.

(Like all things Canary) We wanted something easy to use that delivers immediate value. Here we present several new approaches, and look forward to hearing from the community on the usefulness and ways to increase insight here for network defenders.

Challenge: 

APIs are everywhere and permeate most organization’s daily web based workflows. Both internal and external services often rely on the use of REST APIs. From workstation management to web applications, from complex business logic and application integrations, to payment processing services, APIs form a backbone for all kinds of crucial services. 

It started us thinking, how we might be able to create and use Canaries and Canaryokens to catch or detect unauthorized REST API endpoint or key usage? Our end goal would be for teams to receive an alert when an API key, or tokened URL endpoint they want to monitor, is used by an attacker.

(The AWS API-key is already our most popular Canarytoken, with hundreds of thousands deployed world wide, but we wanted to mint API keys for our own internal services and have them be as useful)

Solution:

At its core, a REST API call is an HTTP request. What we need is a way to receive a request, log relevant headers or parameters and HTTP methods, then send an alert to the Canary console.

We can do that.

There are already a number of HTTP request related primitives built into Canaries and Canarytokens. We can easily leverage these to create a REST API Canary endpoint. This allows defenders to quickly gain insight into unauthorized use of REST APIs within their organization with low setup costs.  

We will show three types of REST APIs you can consider using:

1. Using the simple Web Bug Canarytoken.
2. Using the (fake) Webserver on a Canary.
3. Creating a Custom Canary TCP Service.

Setup One : Web Bug Canarytoken 

This is the most basic form of the request, create a Web / URL Canarytoken, then leave a hint to its use in a document someplace and see if anyone ever runs a script or curl command to trigger the alert.

We will create a Web Bug Canarytoken, annotate our Token Reminder, for later. We will then use PostMan, an API testing tool to send a sample request to start out. 

First we will create the Canarytoken, either in our private Canary Console or at canarytokens.org. 

Notice the full/returned token-URL is:

http://canarytokens.com/traffic/ww0mayx9rro4h2l3keazj0zdk/contact.php

Next, we will modify the generated URL slightly to make it appear more like a REST API endpoint. 

This ought to do the trick:

http://canarytokens.com/api/v1/users/list/ww0mayx9rro4h2l3keazj0zdk

Remember, this URL is just an example. Apart from the hostname and the actual token (in red), you can change all other parts of the token-URL.

Test with a curl command

curl http://canarytokens.com/api/v1/users/list/ww0mayx9rro4h2l3keazj0zdk

Or send requests and track responses with a tool like Postman 

This generates the expected alert:

This is a simple example to showcase the captured HTTP request, source IP address and the User-Agent string. We can see the utility of a quick alert or signal that someone finds this url interesting (and by extension that someone had access to the place it was stored).

However, it is pretty clear by the hostname, that we are hitting a Canarytoken URL. Teams using our commercial Canary offering have the ability to use a custom domain. If you are using the free service, you can abstract away the Domain Name by using a DNS CNAME record.

We create a CNAME record in a domain we control. We want api.example.com to resolve to canarytokens.com. We add a CNAME record to point api.example.com to canarytokens.com, so that when the request is made, we still receive the alert!

Test:

curl http://api.example.com/ww0mayx9rro4h2l3keazj0zdk/api/v1/endpoint

Alert:

We can see that we receive the same alert, even though the hostname was changed. 

Setup Two: Webserver On Canary

Canaries have a built-in feature that allows you to configure a Webserver on it with a single click. With another click you can choose one of the default HTTP Skins. This is a great way to create an enticing server that attackers may attempt to interact with. Below is a basic example of applying the Jenkins Login HTTP Page Skin. Enable Webserver settings in the Canary Console:

When someone browses to the page they will see this.

(By default this exists to capture the credentials an attacker enters – and lets you know that somebody is messing around where they shouldnt be). A Skin here is simply a set of files, styles, scripts and images that you want to present to a visitor to the page. You can also upload custom skins to create any style you want. But! Even having a basic Canary-website like this allows us to capture REST API request attempts. You can post a reference to the service and usage. This could be in Github or other places with script snippets or command documentation. These are basic examples, but we think you get the idea. Below is a modified example of a REST API that allows you to transfer money between accounts. Place this example near the Accounting or Finance developers and lets see what happens.

Test:

curl http://srv02.example.com/api/v1/transfers \

  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \

  -d amount=530 \

  -d currency=usd \

  -d destination=acct_acdcD82eZvKYlo2C8675309 \

  -d transfer_group=ORDER_66 \

  -v

Alert:

When anyone attempts to use this command, we will get the following alert!

In the above example, we only see the basic attributes of the request. (Canary won’t log the request

parameters by default). To capture the POST parameters, we add a custom file. Then alert would include

more context: how much money was attempted to be transferred and to which account!

For more detail on these parameters see this blog post about custom webroot configurations. In order to capture the POST parameters, we created a Defined Skin, you can read further about this here, Capturing Incident Data on POSTs.

Setup Three: Canarys Custom TCP Service 

In addition to the built-in HTTP Page Skins, Canaries allow you to create a Custom TCP Service that listen on a port you define, and gives you control over various interactions. This allows you to mimic services that may reflect websites or a REST API without using the standard HTTP service (80,443). If you would like to learn more about enabling and configuring a Custom TCP Service, be sure to review be sure to review our Service Configuration steps.

We can expand on the basic examples above.

We enable the Custom TCP Service on port 8443 in the Canary Console.

To make sure we don’t easily alert for false positives, we can configure the service to only alert when

TCP Connections send a string we are expecting by adding it to the Alert String field.

Test:

curl -X POST \

-H ‘X-Auth-Token:15dd7c486d81899f64643d6618c47a4efeedacdc’ \

http://srv02.example.com:8443/api/v1/resetpwd

Alert:

As we can see, our alert was triggered when the X-Auth-Token string in the Header matched our

configured Alert string. The Alert string is not required, if you are interested in alerting on any

connection attempts and logging the data sent, then leave the Alert string setting off.

An additional benefit of a Custom TCP Service is we can create tokens that are context specific.

The request URI, consists of: {URI-scheme} :// {URI-host} / {resource-path} ? {query-string} .

Given this is the case, we could create query strings in the request that identify the token.

curl -G -H  “Authorization: Bearer 15dd7c486d81899f64643d6618c47a4efeedac” \

http://srv02.example.com:8443/api/v1/resetpwd/subscriptions \

-d “api- version=2014-04-01-preview” \

-d “location=ThisisinStevesAzureADAuthenticationscript”

We never had to create a specific token, we can infer from the Request the key location dynamically. We don’t have to be as obvious in the parameter value either. We simply place these examples in links or in scripts. When the request arrives we receive an alert with the complete HTTP Request.

Placement:

One last topic needs to be considered here. How would an attacker find these examples, and what might

entice them to use them? That is a great question. There are a number of places where details for how to

connect to a REST API might show up. Much like opening a document with a Canarytoken or a URL

Canarytoken. We want to find ways that would cause someone to interact with our fake REST API. Here are a few thoughts and ideas. We would love to hear more about what you come up with as well.

Idea One: Place the REST API Canarytoken example in an Internal code repo, disguised as a  comment,

or a reply or a code snippet. 

Idea Two: Place a link in an internal page that calls out to the specific URL, or a file share, or laptop,

that looks like a developers workspace. There are a number of additional references and resources that

may also be useful to think about how your organization may find helpful.

We found the blog below from Detectify to be a helpful resource to get teams started thinking about how

an attacker might find and use a REST API endpoint or keys they discover.

Source:  https://labs.detectify.com/2021/08/10/how-to-hack-apis-in-2021/

Conclusion 

We can leverage already existing Canarytokens and Canary HTTP services to receive and alert

on unauthorized REST API request attempts. With the already existing HTTP primitives teams can extend or embed these examples within their organization and applications to detect and alert on unauthorized use.

References:

API Examples:

https://docs.github.com/en/rest/billing#about-the-billing-api

https://api.nasa.gov/

https://stripe.com/docs/api

https://docs.microsoft.com/en-us/rest/api/azure/

https://betterprogramming.pub/the-anatomy-of-an-http-request-728a469ecba9

https://labs.detectify.com/2021/08/10/how-to-hack-apis-in-2021/

Hacking APIs ,Breaking Web Application Programming Interfaces – by Corey Ball.

The AWS API Key Canarytoken (paid and free) is a great way to detect attackers who have compromised your infrastructure. The full details are in a previous blogpost, but in short: 

1. You go to https://canarytokens.org and generate a set of valid AWS API credentials;
2. Simply leave those in ~/.aws/config on a machine that’s important to you
3. Done!

If that machine is ever breached, the sort of attackers who keep you up at night will look for AWS API credentials, and they will try them. 

And when they do, we let you know that you’ve been breached.

When you receive an email/SMS/Slack message letting you know that the AWS API key that you left only on BuildServer-7 in Server-room #12 just got used to login to AWS, you know you have a problem. 

The underlying Canarytoken infrastructure relies on AWS APIs logging their own execution to CloudTrail. This lets us identify: which IP made the call; which API was executed (including both the service name and function); plus other details about the client executing the call.

The effectiveness of the AWS API Key Canarytoken shouldn’t be underestimated. Attackers have to try them; they could be the keys to the victim’s kingdom. If they don’t try the keys, they might be missing a golden opportunity. AWS API Keys are about the juiciest bait you can dangle in front of adversaries.

For defenders, the keys we supply are simple and entirely safe. They’re not tied to anything owned by the defender (the infrastructure sits completely at Thinkst), the keys have no permissions so they can’t be used to do anything, and no agents or software needs to be installed. You simply drop a text file, and wait to be notified if anything happens.

Between Canarytokens.org and our commercial Canarytokens offering, thousands and thousands of  machines worldwide have AWS API Canarytokens lying in wait for attackers.

Can AWS Canarytokens be detected?

This introduces a new goal for the super stealthy attacker. If they find an AWS API key on a server, can they tell if it’s a Canarytoken (or if it is a legitimate key?). Our view is that this only matters if an attacker can determine this without triggering the token. In other words, the Canarytoken fails if attackers can come across API credentials, and perform some test that returns whether or not the credentials are Canarytokens but defenders are never notified of the test. 

The key (heh) to making this happen as an attacker, would be to find AWS APIs that don’t log their execution, and also reveal information about the calling API Key to the caller.

For our purposes, there are four classes of error responses from AWS APIs:

1. Logs to CloudTrail, reveals no information about the API key
2. Logs to CloudTrail, does reveal account details from the API key
3. No CloudTrail logs, reveals no information about the API key
4. No CloudTrail logs, does reveal account details from the API key

As defenders,  4 is the worst case. If an API fulfills number 4, then Canarytokens can be detected. 3 is not great either, but softens the impact to “attackers can determine where credentials are valid, but not whether they’re Canarytokens”.

The next bit may surprise those who’ve never worked with it: the AWS API is a mishmash of response codes, error code, error strings, exception names, and data formatting. It is anything but consistent.

In the past, it’s been the case that a tiny subset of API errors did fulfill the constraints for class 4. The folks at Rhino Security blogged about this (RIP Spencer). While AWS (very slowly) does seem to react to reports on APIs which don’t show up in CloudTrail, that’s not sufficient. As it stands, we’re not aware of class 4 error responses currently in the AWS API, but at the rate at which new APIs are added, that’s little solace for thinking about the future.

There are certainly class 3 error responses in the current AWS API if you go looking hard enough.

In summary, the AWS API has a small number of endpoints which don’t log their own usage to CloudTrail, creating a blindspot for defenders. What do we do in those instances?

IAM Credential Reports

Scott Piper pointed out that Amazon actually does provide a backup mechanism for identifying credential usage. The IAM service lets you generate and download a report for all your credentials. This is a CSV file where each row belongs to an IAM user, and some of the columns identify when an access key was used, and on which AWS service it was used.

With the credential report, there are no longer class 3 or 4 error responses in a practical sense; we can also check the credential report to see if an API key was used, so no longer need to just rely on CloudTrail.

The report has three main drawbacks.

1. The report can only be generated once every four hours, so at worst there’s a four hour delay between credentials being used, and you seeing the notifications. 
2. The fidelity of the reports is low; you only can tell the last time the key was used and on which service (e.g. ec2, iam, sts, etc). You can’t tell which function was called. 
3. You don’t get any client information, including IP addresses or client versions.

However in spite of these drawbacks, it’s a huge step up in terms of reliability of this token type. Attackers don’t have places to sneakily test their API keys. This safety net means that every key usage will be detected and alerted on, albeit sometimes with lower fidelity artifacts. 

Even if you didn’t know the attackers IP Address, would it be worth knowing that the key that you placed on \\code-sign-22 was used to login to AWS this morning? Absolutely!

In fact, if a key is only used on what was previously a class 3 or 4 API, that’s even more of a signal since it implies the attacker is actively trying to avoid detection.

So we built an AWS safety net, to help us catch attackers who actively try to avoid detection.

Building an AWS Safety Net

The architecture is very straight forward. An AWS Lambda function fires on a periodic schedule. It pulls in the credential report from IAM, iterates over all the rows, and determines whether the last recorded use of the key happened more recently than the CloudTrail logs show. If so, the Safety Net kicks in, and sends out the alert:

Browsing to the history page for the incident, you see an annotation telling you that the Safety Net picked up this usage:

The Safety Net was deployed to our commercial customers a little while back thanks to Jason, and we’ve recently rolled it out to the free users on Canarytokens.org too.

Wrapping Up

AWS API logging previously left a small but significant gap that potentially gave attackers a way to use Canarytoken API keys without triggering alarms. With the deployment of the Safety Net, this gap has been plugged.

To see a World in a Grain of Sand Rice

– William Blake mh

If you are on TikTok (or a fan of talk shows) at the moment then, no doubt, your feed has included coloured rice being tossed in the air in the form of song lyrics, beloved cartoon characters, and even famous faces.

@mr.riceguy

Whilst coloured rice is not a new thing (for most preschool teachers, it is a cheap and effective way to keep kids entertained), a bunch of TikTok-ers have made a living off turning this simple play-thing into a full-on career. And, obviously, when a current trend is well-suited to our logo, we have to give it a go. Here’s how we got there:

What we used

• Rice
  • Our whole logo only needed 500g, however, we needed a few attempts to get it right and ended up using about 3kg. White rice is reusable, a jumble of multi-coloured rice, not so much…
• Food colouring
• Vinegar
  • 1 teaspoon per cup of rice
• Large (relatively stiff) rice tossing surface
  • We used a shelf from our cupboard…

How we did it

• We started by colouring the rice
  • We mixed 4 cups of rice with 4 teaspoons of vinegar and added food colouring until we got to the colour we wanted. The vinegar helps the colour spread evenly and get absorbed by the rice, so that, when playing with it, the colour doesn’t stain your fingers.

• Once mixed well (and to your colour-liking), we then dried the rice on trays lined with paper towel. The thinner the layer, the quicker it dries. Ours took about 2 hours.

• We scaled and sketched our logo onto the board
  • If we had had a projector, this would have been a much quicker + easier process, but alas.
  • Note that the image that you sketch out on the board will be reversed when tossed into the air. We drew the logo on in the correct orientation and then simply flipped the footage in editing

• Practice makes perfect
  • We had a few attempts with random shapes, just using white rice, so that we could practise the flip motion (and then could reuse the white rice)
  • We found the trick was to not raise the board too aggressively, but drop it as quickly as possible

“TEST”

Ultimately…

Verdict

Using the WireGuard Canarytoken

Our Goal

Version 1: a rough draft

Problems to Address:

A WireGate instead of a whole WireGuard

Our goal admits an important simplification: an attacker trying out the WireGuard Canarytoken client config must first initiate an encrypted session before she is allowed to do anything else.

A closer look at the WireGuard protocol shows how this can be done. Although peers on either end of  a WireGuard tunnel support the exchange of a few different types of messages to transfer encrypted payloads and to initiate encrypted sessions, the initiation is a single roundtrip of handshake-messages between the two peers. After just this handshake initiation message, the server (responder) knows which client has initiated the handshake – and in our case who to alert.

The fit with our Canarytoken use is even better with a closer look at the handshake initiation message:

Once the static client public key is decrypted, we know who to notify. If the encrypted timestamp that follows also decrypts, we can confidently say that only a device with knowledge of the corresponding client private key could have produced this message.

A caveat to this, is that the message could have been replayed by a passive observer of the device with the client private key on it. It works in our favour here though that, being a Canarytoken, the client private key is rarely, if ever, used, and that our server can insist on fresh timestamps to reject stale initiations. This gives us a high degree of confidence that whatever sent the initiation message has gained access to the client private key only installed on a single device.

All this can be inferred from just the first handshake initiation message. So instead of  supporting the full WireGuard protocol, we implemented a small “WireGate” service which only supports the handshake initiation message. This simplifies a bunch of the problems from the initial rough-draft:

• There’s no need to null route traffic, because there isn’t any routing happening. Only individual UDP packets are checked to see if it’s valid handshake initiation and everything else can be ignored.
• There isn’t a need to maintain a shared set of valid keys with a separate WireGuard service.
• As the number of keys grows, it’s also much simpler to reason about WireGate’s performance. (The initial handshake decryption is done in relatively constant time so won’t limit the number of keys created.)

With only a partial protocol implementation, it’s necessary to ask if WireGate is realistic enough for an attacker to interact with. WireGate only needs to set off the alert to let us know someone has the client private key to have done its job, but it’s less useful if an attacker is able to trivially detect that it is only a partial WireGuard implementation.

Adding WireGate to canarytokens.org

Canarytokens.org is the free Canarytokens server we host for the world. It’s generated close to a million tokens but this introduces a new complexity for us. Paid customers get their own canarytoken-servers, so we can run separate instances of WireGate per-customer, but the public server is shared with everyone on the internet. The same server key is used in all the Canarytoken WireGuard client configs issued, making for an easy tell. We had to do better.

In the ideal case we’d simply create a new server key for each Canarytoken WireGuard client config. To see why the naive approach doesn’t work, consider the fields in initial handshake packet:

The handshake is negotiated with the static server public key in the Canarytoken WireGuard client config. Without prior knowledge of the corresponding server private key, it isn’t possible to decrypt the encrypted client public key and determine the owner to alert. The naïve approach to try every server key for every client key ever issued as a Canarytoken, would take increasingly longer to handle every handshake initiation message that arrived.

The simple workaround for this is that Canarytokens.org uses a fixed-size pool of server keys. The Canarytoken WireGuard client configs are issued with randomly chosen server keys from the pool. To skip managing 1000s of keys, the pool of keys are derived from a single private key seed.

Decrypting the client key with each server key in turn would work fine, but it can be done much faster. The initiation message includes an unencrypted keyed MAC allowing us to guess and confirm which server key it was encrypted for. For each server key, we derive the corresponding MAC key and verify the incoming handshake MACs with each. This finds the correct server key much faster and only increases the processing time for a handshake initiation message by a constant factor (number of pool keys). It comes down to a few lines in Python.

As the WireGuard whitepaper points out, this is only a very slight weakness in the handshake initiation identity hiding that makes guessing the server public key possible (and not the server private key). It’s only trivially useful here because of the circumstances we contrived: the WireGuard server already knows all the server keys (unlike an attacker who can only passively observe messages). We’d be interested to know if there’s better cryptographic tricks to find some biased bits in the handshake initiation message to build an index to efficiently look-up either server key or client public keys. Our attempts all brushed up against parts of cryptographic primitives designed to resist what we were doing. We like to think that, rather than this being a limitation of our own engineering abilities, it’s a virtue of good cryptography in use by WireGuard being harder to mis-use in ways it is not designed for.

Conclusion

At Thinkst Canary, we’ve been rolling our own partial-protocol implementations where it makes sense for security and performance on lower-resource devices since almost day-1. That said, partial protocol implementations won’t fit every problem as well as WireGate does for WireGuard. If this problem also required emulating network services accessible over the WireGuard VPN, we’d be better off with a full implementation, ideally an isolated emulation. (For meatier protocols where native code implementations are un-avoidable, those run sandboxed.) If it wasn’t clear already, we think WireGuard is great. The more time we spend working with it, the more we’re convinced others should too.

(The making of a MySQL Canarytoken)

tl;dr

Consider this scenario: An industrious attacker lands on one of your servers and finds a 5MB MySQL dump file (say, called prod_primary.dump). What do they do next?

As soon as they do, you get an email/SMS/alert letting you know:

Eds note: You can create and deploy these by visiting canarytokens.org

There are obvious benefits to these sorts of booby-traps, but some rise above the rest:

• • They can be deployed in seconds;
  • They aren’t prone to high false-positives;
  • An attacker who suspects you are using these is no better off for knowing this (if nothing else, they now have to second-guess everything they touch);
  • It’s such a pure illustration of attack-minded defense.

It Begins…

• Can we get the system to surf to a URL we control;
• Can we get the system to lookup a DNS entry that we control;

Forcing a DBMS to reach out to other servers is a classic SQL Injection challenge and past work had laid out techniques [datathief][squeeza] to solve this on other DBMSes (Oracle and MSSQL) by leaning heavily on their robust shell programming abilities. MySQL was different though with less functionality in this regard. On my Linux host (or Docker container), I couldn’t use cute tricks like calling a LOAD DATA INFILE from a file path with a Canary-DNS entry

(e.g., LOAD DATA INFILE ‘\\DNS-entry.thinkst.com’;).

A quick search of attack-minded-blogs seemed to corroborate my thinking, that MySQL was restricted enough to prevent these types of tricks in its default state.

Sizing up the problem

After much fiddling and searching I found a promising lead through database replication.

This was a mixed bag. The community/open-source version of MySQL does support replication but a server would need to be configured (through its config files) to support it. We can’t convince an attacker who stands up a temp MySQL server to first reconfigure their DBMS and I almost gave up. But then in-band-signalling once more saved the day (or led to horrible insecurity). It turns out that these configuration files can be overridden at runtime by SQL commands!

Keep in mind that all we want at this stage is a set of MySQL queries (run on a standard MySQL instance) that will reach out and touch a foreign server somehow.

I set up a netcat listener on a remote server and on the MySQL server side I configured the replication to point to it:

CHANGE MASTER TO MASTER_HOST=’my-server.thinkst.com’, MASTER_PORT=3306, MASTER_USER=’root’, MASTER_PASSWORD=’root’, MASTER_RETRY_COUNT=1;

Then, I typed START REPLICA; and waited.

And waited.. and… nothing. The netcat listener showed no evidence of a connection, and I had nothing on my MySQL session.

Running SHOW REPLICA STATUS; on the MySQL server reveals that the server did try to connect (already now an option for a DNS token), but the connection failed. I replaced netcat with tcpdump and retried the process. This time I did see a connection to my server, but no data was exchanged, and the MySQL status still reported a failure.

A deeper dive into the MySQL Handshake

Turning to the MySQL documentation on how a connection between a client and server is established, we learn that although the client (or replica-seeking server) initiates the connection, once open the client then waits for the server to return a Handshake packet. This packet describes the server’s capabilities and supported authentication plugins to the client, so they can mutually agree on the fullest feature set for the connection.

We had enough capability at this point to create a DNS-based Canarytoken. Ie.

1. Create a unique DNS host-name on a domain we control; (this is done by simply visiting Canarytokens.org and creating a DNS-Canarytoken: we get back something like: 0iep6h5na3p4coxx4hax132b8.canarytokens.com)
2. On the MySQL server, issue the commands MASTER_HOST=’0iep6h5na3p4coxx4hax132b8.canarytokens.com; START REPLICA

Even though the actual replication never takes place, the MySQL server resolves the DNS name of the foreign server (effectively tripping the DNS token) and letting us know that it’s happened.

But… While a DNS based canarytoken (like this one) is great at letting us know that something happened (someone ran this mysql import command) it isn’t great at giving us more details. Our DNS server is able to tell that the record 0iep6h5na3p4coxx4hax132b8 was requested, but can only tell us the IP address of the DNS server that made the request.

We can do better.

If we can get the remote server to complete the MySQL handshake, we’d get a connecting IP address and we can possibly stuff more information from the MySQL server into the username/password fields that we submit.

There were a few useful blog posts that helped describe the packet format, but it was a relatively complex encoding that included some fixed-width fields, some NULL-terminated fields, and some packing fields. Since I didn’t need the connection to complete, I simply captured (above figure) the Handshake packet from a real MySQL server (configured to explicitly not support SSL to simplify the next steps), and wrote a small Python server to send those captured bytes to every connection.

This allows the handshake to go further (and lets us submit a username/password combination from MySQL to the foreign server).

This gives us all the pieces we need to create a point-and-click Canarytoken to cover this use case (and we did). Here’s how you can use it.

Using the Canarytoken

If you visit https://canarytokens.org and click the token selection dropdown, you’ll notice a shiny new MySQL token.

In typical Canarytoken style, you then supply just two pieces of information.

1. An email address to receive the alert;
2. A reminder note to jog your memory when this alert fires;

That’s it. When you hit “Create my Canarytoken”, we give you two quick ways to make use of the token, choose the one that’s best suited for your scenario (likely option 2):

1)

SET @bb = CONCAT(“CHANGE MASTER TO MASTER_PASSWORD=’my-secret-pw’, MASTER_RETRY_COUNT=1, MASTER_PORT=3306, MASTER_HOST=’k5sk4zeo5csej6pps4vkgzmtp.canarytokens.com’, MASTER_USER=’k5sk4zeo5csej6pps4vkgzmtp”, @@lc_time_names, @@hostname, “‘;”);

PREPARE stmt FROM @bb;

EXECUTE stmt;

START REPLICA;

1. 1. 1. Her server will lookup k5sk4zeo5csej6pps4vkgzmtp.canarytokens.com triggering the DNS token and letting us know that the MySQL dump file we left safely on NYC-DC1 was just loaded into MySQL somewhere;
      2. Her server will connect to our fake MySQL server (which now knows her MySQL Server’s IP address, which is a strong thread to pull on);
      3. Her server will attempt to login to our fake MySQL server with the username: ‘k5sk4zeo5csej6pps4vkgzmtp”, @@lc_time_names, @@hostname which now gives us more information on the attacker that we can report on when letting you know the token has tripped.

2)

For users who don’t have a mysql dump file lying around (or don’t want to risk creating one with real data), we also offer option-2, where we take a sample MySQL file, run it through a quick mixer to generate some random linked tables with data, and insert the tokened snippet into it. No mess, no fuss. Simply drop the file on your server (or in your dropbox, or in your email) and if you ever get the notification to let you know that it’s been loaded, you know you have a problem.

In many ways, tokens like this are a joy. If you leave this Staff_Salaries-2021.mysql-dump.sql file in your email and forget about it for 10 years, it costs you nothing.

If you get a notification letting you know that the mysql dump you left in your mailbox just got loaded into a MySQL server in $FOREIGN_COUNTRY? That’s priceless.

There’s no chance of it being an accidental alert, and there’s almost no chance that an attacker would find a large enough mysql dump file without loading it into a DB to check it out.

Absolute win.

Bizzaro Bonus:

Last month Florian Roth (cyb3rops) reacted to the news of Mimikatz dumping RDP credentials by asking how we could easily inject fake credentials into machines.

https://twitter.com/cyb3rops/status/1397440903476883458

Markus Neis pointed out that on Windows, cmdkey allows you to do this: 

https://twitter.com/markus_neis/status/1397472760859856897

Which gives you that one alert, when it matters.

This means that an attacker that compromises this desktop gets bogus credentials [administrator/super-secret-123] but more importantly, they get a pointer to your Canary [\\02-FINANCE-02]

We’re starting to repeat ourselves here, but this gives you that one alert that lets you know when bad stuff is happening.

Would you know if your phone was hacked? Even the most powerful people in the world (if you use wealth as a proxy for power) don’t.

So when you notice that your CEO is still in Seattle, but his VPN was just accessed from Saudi Arabia… that’s a raging clue.

We love Canarytokens like the WireGuard token because even if the attacker suspects that it’s a trap, it’s really hard to avoid using it.

What do you do as an attacker? Simply ignore the potential access it might yield? (You could, but that’s going to slow you down terribly,  which would be a win for team defense too).

The WireGuard Canarytoken is available on all Canary servers today and will make its way to the public Canarytoken server in the next few weeks (when we will explain more of its inner workings).