• Canarytokens.org welcomes Azure Login Certificate Token


    The AWS API key Canarytoken is a perennial favourite on Canarytokens.org, and we’ve heard requests for a similar token for Azure.

    In this blog post, we introduce the Azure Login Certificate Token (aka the Azure Token) to Canarytokens.org1

    As with all tokens, you can sprinkle Azure tokens throughout your environment and receive high fidelity notifications whenever they’re used. Place one on your CTO’s laptop, or on every server in your fleet. When attackers breach that laptop, or servers, or machine, they’ll search for useful credentials and discover the Azure tokens. Such juicy credentials are too tempting to ignore, and when they try them, you’ll be alerted to the compromise.

    Why is the Azure Login Certificate Token useful?

    Azure is second largest provider of cloud infrastructure services in the world. Hundreds of thousands of organizations use Azure Cloud to run their infrastructure. Thanks to the growing Infrastructure as Code movement, many of them are bound to use programmatic command line access to manage their infrastructure.

    Attackers know this too.

    Searching for Azure credentials is almost standard post-exploitation behaviour and finding login certificates are an attacker’s dream. This token turns that around. One alert, when it matters.

    Are Azure tokens just useful to Azure customers? Of course not; Canarytokens are useful across your actual vendor lines. You don’t need to be an AWS customer to deploy actionable and useful AWS API Key Canarytokens, and nor do you need to be an Azure customer to find Azure tokens useful at detecting compromises in your network.

    Attackers who find them won’t decide not to use them because they really don’t think you are an Azure shop. They will lick their chops while testing access (and in doing so will tip their hands).

    Creating Azure Login Certificate Token

    It’s dead simple: head over to canarytokens.org, our public Canarytokens service:

    1. Select ‘Azure Login Certificate’ from the drop down list.
    1. Enter an email and a token reminder. We use the email address to notify you when the token is tripped. The reminder you choose will be attached to the alert. (Choose a unique reminder! Nothing sucks more than knowing a token is tripped, but being unsure where you left it). A good Reminder is something like “Azure Token deployed to c:\Users\Administrator\ on DC-LON-02”, which clearly highlights where you placed it.
    2. Click on “Create my Canarytoken”:
    1. Congratulations, your new Azure token is ready to be deployed! The output displayed can either be copied into a new file in the place you want to deploy the token, or you can download a file and move it into place. Don’t forget to delete an intermediate copies of the data.

    Testing the Azure token

    On Linux (with the az tool installed), the token can be triggered simply by running:

    $ az login --service-principal -u <app-id> -p <password-or-cert> --tenant <tenant>

    with the relevant parameters updated using the information from the token. As an example:

    Within 5 to 10 minutes you’ll get an alert notification indicating that the credentials have been used:

    Where to deploy the Azure Token

    Place the Azure token config file alongside the certificate in a juicy place for potential attackers to find.

    Most systems have a ~/.azure folder (much like the ~/.aws or ~/.ssh) and you can place the config file and certificate there.

    Behind the scenes

    The route followed is similar to what we do with AWS Key Canarytokens. In short, we pre-generate credentials programmatically into a pool of available credentials because each takes a few seconds to create. These are later allocated to a user when they request a new Azure Canarytoken. We then monitor the usage logs of the Azure accounts in which the credentials were created, and if credentials were used we send the alert. 

    Wrapping up

    The new Azure Canarytoken gives Azure customers (and everyone else) a simple new way to detect breaches, by deploying Azure credentials at no risk to themselves in places attackers would typically find them. 

    Azure tokens are currently live on Canarytokens.org, and (as always) are completely free.

    They take just minutes to create and deploy (even if you are a slow typist). Try them. They are totally worth the time.


    1Commercial customers have had this token for a little while now.

  • Swipe right on our new credit card tokens!

    Swipe right on our new credit card tokens!

    Detect breaches with Canary credit cards!


    Today we’re releasing a new Canarytoken type: actual credit cards! 

    1. Head over to canarytokens.org;
    2. We give you a valid credit card (number, expiration, and CVC);
    3. If anyone ever attempts to use that card you’ll be notified.

    We recommend placing one anywhere you store payment information. If you ever get an alert on it, you know that that data-store has been compromised.


    Canaries generally aim to look like something an attacker would want to interact with. It’s why our mantra has always been that Canaries should look valuable (instead of just vulnerable). Historically, these have been network services, or a juicy repository of sensitive information that usually would encourage an attacker to advertise their presence as they move through the network attempting to find firm footholds. Canarytokens expand on that to include files or data that reliably trigger alerts when accessed. 

    Our new credit card tokens fit this bill perfectly. We give you a perfectly valid credit card. You store it somewhere and if it’s ever used, we will let you know.

    Mix it in with your store of saved card data or on payment gateways. An attacker who plans to test the cards (as they normally do when obtaining them) or attackers who try to use them will immediately advertise their presence, and your response team can spring into action.

    Using the token

    Using this new token is easy, just head over to Canarytokens.org, and select Credit Card token from the dropdown.

    Then enter the email address or webhook URL where you want to be notified when an attempted transaction occurs (we never use this to spam you or sell you things, it’s only to notify you when this card is used):

    Hit “Create my Canarytoken”, and after a few seconds we will give you a set of unique, valid (real) credit card information, complete with generated name, card number, expiration date, and CVC:

    You can also download this information as a CSV to programmatically import into your storage location. 

    Some places we recommend putting these include:

    1. Databases where you store customer payment information
    2. Email inboxes (PSTs) to get an alert on email compromise
    3. If you’re concerned about an insider, put one or two in a Word document on an internal file share in a file called something like: “travel payment info.docx”

    Take a deep breath and relax, the hard work is all done!

    If someone does try to use the card, the transaction will fail, and you’ll get an alert like this in your email with the merchant name, the amount of the transaction, and the note you put in when you created the account:

    This is a high quality alert–someone is actively trying to monetize data that they should only have been able to get from wherever you put this token. Like all other well deployed Canarytokens, it also self identifies. You can drop one in each payment store or database and forget about it (at least until the card expires). When you get the alert, you will know immediately that it’s the credit card from the Lisbon DB that was used, and you know immediately where to start investigating.

    The chances of a false positive for this alert is close to nil and historically it’s been clear that the quicker you are able to react to the compromise, the more you are able to contain the splash damage of the event.

    Conspicuous deception

    Canaries and Canarytokens have caught red-teamers, fast-fingered insiders and full-blown attackers all over the world. We expected them to when we started Canary. What we didn’t quite expect, was the deterrence factor once attackers became aware of their presence. Last year, during an external red-team engagement, we placed attackers on a presentation laptop in our conference-room. The attackers, knowing our proclivities, were afraid to move beyond that system, paralyzed (almost into inaction) for days. This matches feedback we’ve received both privately and publicly for years:

    We’ve been noodling on this a little bit and we’re calling it conspicuous deception. Letting people know you are running Canaries or Canarytokens in order to alter their behavior.

    We think the credit card Canarytoken is a good example of this.

    If this token has the impact that we hope, savvy attackers, or the buyers of their stolen dumps, will have to start considering the risk of a test swipe destroying the entire set. As merchants and their payment processors leverage this new visibility, they can respond to a test swipe event much more quickly, and with better understanding of the potential splash damage of a breach. Typically credit card companies and banks identify breaches through analysing multiple reports of fraud looking for commonalities in their transactions (such as physical charge locations, websites where the card was used, or payment processors that were involved). This takes time for sufficient fraud reports to flow in before the breached location can be identified. This token allows for near instantaneous identification of a breach.

    For low-tier attackers that continue to breach and steal cards without changing their tactics, this token will reduce their ability to monetize and commit fraud. Savvy attackers may start looking for patterns in the bank identification numbers (BINs) that we issue, and proactively deleting or excluding them from their dumps. For this reason we are in discussions with a number of banks to onboard their BINs to the system too, further mixing in legitimate cards with tokens. 

    It’s a compelling argument: “Would you like attackers to first remove your bank’s cards from dumps they steal?” 

    The more BINs we can cover with tokens, the more deterrence the token provides – even to organizations that have not deployed these tokens to their environment. This is a benefit of conspicuous deception, the possibility of the dump being tripwired provides protection even if it isn’t actually seeded with tripwires.


    Canaries and Canarytokens are powerful tools that are easily deployed. Recently a security researcher, Daniel Hückman discovered his AWS Canarytokens stored in his CircleCI environment being improperly used. 

    The credit card Canarytoken provides more ways to monitor your environment, as well as the exposure to your data by third parties. Credit card fraud amounts to almost $40B per year worldwide, we hope that with faster response times to a breach to help make a [small] dent in that figure.

    We think that our Canarytokens offer great protection and detection capabilities while being easy to deploy, and cost-effective (free!). By giving them away for free, we introduce a risk for attackers who are trying to monetize their access, from AWS credentials that may provide access to the crown jewels to an Excel document called “2022 Taxes”–attackers need to step a little more carefully. 

    We hope you’re as excited about this new token as we are (and that it never has to alert you).
    Ps. if you are a bank/card-issuer that wants to work with us to help protect your customers too please drop us a note at research@thinkst.com

  • Seasonal themes, delighting users & small UX touches

    Seasonal themes, delighting users & small UX touches

    We’ve written before about the effort we put into UX choices in our app. We don’t consider problems solved just because we kicked out a feature in its general vicinity and we are super strong believers in “small things done well

    This came to the fore again recently when we included a “seasonal theme” into customer Consoles and I figured it was worth a brief post to examine our thinking around (even) short-term UX.

    In our early days we’d give a brief nod to seasonal changes by slightly altering our Twitter avatar.

    Having an actual legit designer on the team gives us significantly more leeway, and so “Console Seasonal Themes” was born. The plan is to (very infrequently) add small non-obtrusive splashes within the customer Console when a reasonable opportunity arises. These splashes should be subtle and hopefully bring a quick smile to someone’s face as they go about their day.

    Halloween was the first opportunity to take it for a spin, and Blake threw up a few concepts. We wanted to make sure that the theme never came close to interfering with anyones workflows so ended up with two main areas to decorate: the Console header, and the Canary logo. He chose to animate ghostly birds in the header, and the logo got an appropriately spooky makeover.

    It was beautiful but before unleashing it on all customer Consoles, Nick felt strongly that we should also include a button to disable the animation/effect.

    We know how some customers prefer to even disable minor window animations in the UI so it was the right call. I was originally ok with simply feature-flagging the effect, which would allow the CS/support teams to disable it if a user complained, but Nick correctly pointed out that this was making the customer do too much to switch off something they never asked for.

    So he added a toggle under settings to disable the animation:

    This seems like a reasonable place to ship it, but there is still a flaw: how does a user who signs in know that the animation can be turned off in settings? If they don’t know the toggle is there, it might as well not be.

    So the team next tried adding a small unobtrusive button on the actual header.

    The generic ℹ️ button looked like a candidate for replacement. We went through a few quick variations and settled on a relatable jack-o’-lantern:

    The button that only shows up as you approach the click-zone is a pattern we use elsewhere in the app to keep the interface clean while offering needed functionality but once more the problem would be: how does a user know they should go there to float up the button?

    Instead of ever making the button completely disappear, we opted to use colour and a slight wiggle to bring the user’s eye to it:

    At this point, one would be tempted once more to ship it, but if we are being completely honest, the current jack-o’-button, even with its little wiggle could slip by as part of the animation. It isn’t immediately obvious (unless you’ve spent as much time as we have looking at it) that the lantern is actually a button.

    So what we wanted was a button that didn’t look like a button but that people would know was a button, and we wanted it to be well hidden enough but not too much. These kinds of seemingly opposing constraints aren’t atypical in UX. Folks can think in terms of trade-offs or middle-ground, but with enough effort, we usually end up with a better result than just meeting-in-the-middle.

    The final form then was wiggling, colour-changing-jack, but with a quick flash when the page loads to let you know that something was under there. It didn’t matter if the text escaped too quickly, because your natural reaction would be to hover in the area (which could then activate the button).

    Ultimately, it was worth it. We enjoyed making it happen and customers (at least some of them¹) found it delightful!


    ¹ 153 Consoles disabled the effect (Which is less than 10% of those it ran on). Counter-intuitively, we count that as a win too, since we gave customers that option and they clearly found the button!

  • Company Lessons (from YouTubes “Hot Ones”)

     I recently discovered “Hot Ones” on YouTube. If you haven’t seen any of the episodes, you should (because they really are fantastic). This isn’t really a controversial opinion: their YouTube channel has 12 million subscribers and almost 2,6 billion views.

    The show has a few lessons that I think are worth noticing/stealing. I’ll discuss 3 of them here (even if they are kinda random).

    1) Genuine Warmth

    One would expect the show to lean on a kinda gotcha-slapstick routine: we all laugh at celebs suffering, but that isn’t really how it goes. The host (Sean Evans) suffers every moment with his guests and is super empathetic throughout. It’s not adversarial at all and guests reaching the end have a kind of shared bond with the host (and the audience).

    Lots of companies are polite to customers (in front of them) but snicker about them behind their backs. Everyone recognizes the meme of tech-support insulting “idiot users” who just don’t get it. Even if this stuff isn’t said directly, it can’t really be hidden. It eventually seeps out of an organizations pores.

    This is one of the reasons we focus so hard internally on customer love. We aren’t just doing it to be polite. We know that our customers could have bought anything, but they bought us. We deeply want them to win and we deeply want them to get it. This also can’t be hidden (and also seeps out through our pores).

    Putting in the work (off camera)

    Sean Evans is an incredible interviewer. There are a bunch of compilations online of guests being impressed by the depth of his questions.

    Part of this is because the hot-sauce takes out guests ability to answer:

    But the questions are definitely deep and super well researched:

    Interviews in the past few years explain how they get this right:

    Between episodes is when the real work begins…

    Once a guest is booked, the three-person research team goes to work. “There’s a lot of armchair psychology that goes into the show,” says Schonberger. In practice, that means Evans’ brother — Gavin, who lives in Chicago — will compile a dossier. “He’ll basically read like every article there is, every profile, every Reddit AMA, like reading everything that he can find and create,” says Schonberger, noting that they can run to something like 30 pages long. “It’s almost like we’ve created our own Wikipedia template that’s suited to the show.”

    Sean, on the other hand, does the videos. He’ll consume 12-24 hours of clips, looking for breadcrumbs. …

    Schonberger does some of the podcasts, which means he listens to everything he can get his hands on. It’s the same idea as with the videos, except people are usually less guarded on podcast appearances than they are during video shoots. Then they compare notes, and Evans and Schonberger come up with ten topics to hit during the interview.

    the verge

     We’ve been championing this line of thinking forever. Boxers don’t win their fights in the ring. It’s why we push so much for putting in the miles. It’s why we spend so much effort working on the tiny details that most people won’t see. It’s because the results are worth it (even if most ppl don’t know what’s behind it).

    Following a Script (This is an unusual / subtle one)

    When a new person joins us in customer-success/pre-sales we teach them about Canary, why we build it and how we demo it. One personality archetype quickly decides that they will learn the product but will wing-it for demos. They seem slightly surprised that we stay close to a fixed format even if we have been demo’ing Canary (or building it) it for years. Following a script almost seems like an insult to them:  they have a mic and they are smart and they can show off the newest features. 

    With all the praise (in the previous point) for Sean, and now 18 seasons of the show under his belt, it’s worth checking out this quick super cut of guests starting and then clearing the final wing:

    Once it locks in, he almost never deviates from the script. We’ve already established that he isn’t lazy, so why do they do it that way? 

    The same reason we do. Because it works. There are places to customize, and places to go deep but in key areas, it runs on rails. The script works and they’ve stuck with it for 18 seasons.

    We understand why some new people don’t want to demo close to the original script: they are smart enough and skilful enough to not need guide-rails. It’s also human nature that after you’ve done a demo n-times, you want to do the n+1th differently. 

    When you showed 10 people the basic features of the system, you somehow expect that the 11th person needs to see the advanced features because you already spoke about the basics 10 times, but this is a fallacy. The 11th person is also seeing the product for the first time and they may smile through the advanced features – but you aren’t giving them the same experience you gave the first 10 and theres a good chance you leave them confused.

    The show is worth watching – you should check it out…

    [All this was to allow me to watch tons of YouTube videos calling it management-research]

  • Sensitive Command Token – So much offense in my defense

    Sensitive Command Token – So much offense in my defense


    Many people have pointed out that there are a handful of commands that are overwhelmingly run by attackers on compromised hosts (and seldom ever by regular users/usage). Reliably alerting when a user on your code-sign server runs whoami.exe can mean the difference between catching a compromise in week-1 (before the attackers dig in) and learning about the attack on CNN.
    Introducing our new Sensitive Command Canarytoken.

    This quick/simple Canarytoken alerts you any time your chosen command is executed on a host.

    For example: This token creates registry keys to alert you anytime whoami.exe runs on a host.

    If an attacker lands on the server and runs whoami.exe (as most attackers almost instinctively do) they get the results they expect.

    And you get an alert to let you know that something bad is afoot.

    Why this token?

    In nearly every ransomware report, we can see attackers running a series of predictable commands on endpoints.
    Some of those commands are almost never run by regular users through regular usage. This means that they become high quality markers of “badness”.
    Wouldn’t it be great to get an alert the moment someone runs WMIC on your code-signing server? Or the moment someone runs whoami.exe on your SQL-Server? Many organizations will use EDR (Endpoint Detection and Response) tools to do this. However these complex telemetry streams, and detection logic may not be available to small organizations, may not be comprehensively deployed and often require specific configuration changes to alert on these invocations.
    We want to know, and we want the setup to be dead-simple.

    tl;dr – Just show me the token

    If you visit https://canarytokens.org and view the list of available tokens, you will notice a new Canarytoken type: “Sensitive command token”
    Choose the Sensitive command token.
    Next, like other Canarytoken configurations, we add some quick details:
    [1] Where to send the alert, (email or webhook, or both);
    [2] A reminder note, that will help you react when you receive an alert from this token;
    [3] Choose the process that you want to monitor execution for (e.g., wmic.exe, whoami.exe, klist.exe, nltest.exe).
    The Canarytoken server will create a .reg file for you to download and import onto a Windows 10, or Windows 11 system (or use GPO to deploy across multiple systems).
    Once you have downloaded the .reg file, you can import this to the system (or systems) you want to monitor.

    reg import

    Remember this will require local Administrator permissions.

    That’s it! If a user on this host ever runs that (sensitive) command, you will receive an alert!
    It’s worth noting that this Canarytoken does not impede or alter the original command in any way, it simply sends you a near real-time alert that someone has executed our tripwire.
    In the alert you will see the details of the time of the alert, the user that executed the command, and the computer name the command was executed from.
    A subtle benefit of how this token is designed allows you to deploy the same token on multiple machines without making any changes. Running reg import across a host of machines will tripwire them all, and when an attacker runs her command on any one of them, you will know.
    There you go!
    We can now create a Canarytoken to alert us on Sensitive Commands that execute.
    These commands may be sensitive due to the fact that they are rare, used by attackers, or are sensitive to your organization. How you choose to pick the executable is up to you.
    Bonus: the executable does not need to be present on the system. Suppose your organization never uses adfind.exe.  You could add this tripwire, so that if someone ever downloaded that to your system and executed it you would be notified.
    Additionally you sometimes don’t want to run AV/EDR on some machines, or 3rd party systems that won’t allow you to install software, but here you can monitor an executable or five.
    That is all you need to get started using this new Canarytoken. If you would like to learn more about the architecture and mechanics under the hood: continue reading.

    Under the hood

    Problem Statement:

    We wanted to see if we could find a way to generate an alert to our console when a suspicious command is executed on a system. Without interfering with the original command execution.
    The best candidate command for this alert is one that is rare, short lived, and used by attackers.
    In short, if you want to receive an alert if someone runs wmic.exe on a workstation, this may be the Canarytoken for you.  This token creates a .reg file with three registry entries that you can import to the system. You will need Windows 10 or 11, and Administrator access to the systems you want to add this to.

    Token Architecture:

    Our team spent a number of cycles trying various experiments and approaches to generating an alert based on process execution. From doskey, to wmi, to performance monitoring. We settled on an older Pentesting Persistence trick that in the end met all our requirements.


    In his blog post on “PERSISTENCE USING GLOBALFLAGS IN IMAGE FILE EXECUTION OPTIONS“, Oddvar details how to persist in the registry and execute a command on a Windows system that runs in the background. This technique meets our requirements exactly. We needed a tripwire that would run in the background, not interfere with the original process, and allow us to make a custom callback/alert over DNS to alert a system administrator, or security team, that a suspicious command had been executed on the system.
    (We have also seen a debugger variation of this technique used by high profile attacks as well.)
    “The attackers achieved this by having the SolarWinds process create an Image File Execution Options”
    We will repurpose the mechanics of this attacker technique and turn it into a defender tripwire!
    Oddvar demonstrated his technique  to establish persistence and run an arbitrary executable, we will use it to reliably generate a remote alert.
    This Canarytoken does not add any new executable to your system thanks to the wonders of Oddvar’s technique, built-in tools and the Canarytoken server. We will be using Microsoft’s built-in SilentProcessExit monitoring, documented here:
    The specific registry settings we are interested in:
    Image File Execution Options\{ProcessName}
    You will see in the registry output above, we set this to the value 0x200. You may need to ensure that the process you wish to monitor isn’t impacted by this setting.
    Next we set the ReportingMode to 0x1.
    “When silent exit is detected, the monitor process (specified in the Monitor Process box) is launched.”


    The bulk of the work for this alert is what we configure in the MonitorProcess value.
    Let’s break down the command to trigger this alert:
    ""MonitorProcess"="cmd.exe /c start /min powershell.exe -windowstyle hidden -command \"$($u=$(\\\"u$env:username\\\" -replace('[^\\x00-\\x7f]|\\s', ''))[0..63] -join '';$c=$(\\\"c$env:computername\\\" -replace('[^\\x00-\\x7f]|\\s', ''));Resolve-DnsName -Name \\\"$c.UN.$u.CMD.g6jjwnaukbfddgz51kh8tacdc.canarytokens.com\\\")\""
    The various \\\ sequences are to ensure proper escaping for both the command to run and to import into the registry.
    First we call “cmd.exe /c start /min”. We do this as a trick to make the execution of our alert appear to run in the background. Even when we set -windowstyle hidden, with PowerShell, a window would appear, briefly but this wasn’t acceptable to our goals. We also added some code to remove the non-unicode characters and to remove spaces. To ensure the alert data can be passed over a DNS query.
    Next we run a PowerShell command to resolve our Canarytoken DNS.
    We can leverage our ability to encode data into the DNS request. This is documented here
    Our lookup captures computername.username.token. Why PowerShell? We had some interesting collisions with trying to use the %computername% environment variable in say an nslookup command, due to the fact that MonitorProcess parses the registry key and uses the %c to report a status code. So after a few iterations, and experiment we found PowerShell the easiest way to collect the variables we need and add them into the alert.


    • Installing the .reg file requires admin privileges;
    • Selecting the wrong “sensitive executable” will yield false-positive alerts;
    • Hostname and Username are sent in clear-text in the alert;
    • Process level alert: we only see that the process was executed, this approach does not give us command line granularity;
    • These commands may, upon installation, look like attacker commands . Not the alert itself, but the installation, this is due to the fact we are repurposing an attacker trick;
    • This alerts on all executable matches for the filename. If you need further refinement, you can customize this by setting the FilterFullPath registry key;
    • Image Architecture, these keys were tested for x64.  You may need to customize the settings if you are hoping to alert on x86/32 bit execution. C:\Windows\SysWOW64\whoami.exe may evade the alert. Again, something to be aware of if you are looking into these for critical areas;
    • This alert won’t catch a renamed binary since the alert triggers on the process name.
    For example, our Canarytoken alert won’t fire if the binary has been renamed, but that is ok. If you need a tool that alerts on Process Execution, by OriginalFilename, Location and Command Lines, there are several to choose from.  We just wanted to call that out, so you are aware of some of the limitations of these types of alerts.
    We do also see utility for teams to set tripwires for executables they never wish to see executing. Suppose we want to set a tripwire for mimikatz.exe.  We can create the registry key, even though mimikatz.exe is not present on the system.  Then if anyone ever executes a file with that name, we will receive an alert!  You may also extend this to internal tools or executables as well.

    Choosing a binary to monitor:

    Candidate sensitive commands:
    • Short lived, exits quickly
    • Low frequency
    • Low execution prevalence
    • Indicative of suspicious or unusual activity.
    Each organization will need to look at command frequency, look into various threat reports and make that decision. If you need to disable a Canarytoken like this, you could for example set the Image File Execution Options \GlobalFlags value to 0x0, then set it back to 0x200 if need be.
    For example: We’ve seen in nearly all ransomware attacks, schtasks.exe be leveraged. This binary at first blush seems like the ideal candidate to token but  it turns out that schtasks runs frequently in the background, so it doesn’t meet the candidate criteria of Low frequency.
    Additionally Windows will create an event in the Application log when these events occur.  These may allow teams further investigation data. The only requirement to record these events is that the Image File Execution Options\{ProcessName}\ GlobalFlags key be set to 0x200.
    For those interested in additional details,  you will also see the EventID 3000 in the Windows EventLogs to indicate Process Exit Monitor has occurred. Teams that are interested in a lightweight process monitoring record. Can simply create and set a GlobalFlags == 0x200 , and they can observe over time whether or not the process is frequent or not. Also they can determine if the process runs and ends quickly. This may also be a good Indicator of Compromise , for teams not expecting to see a particular process with this flag set.
    As an example you can see below a quick way to search for these events and extract the Usernames.
    $Events = Get-EventLog -LogName Application -InstanceId 3000 -Source “Microsoft-Windows-ProcessExitMonitor”
    $Events | %{ $_.Username }


    We looked at creating a new Canarytoken type that allows us to receive an alert when we specify an executable name that we want to watch for. We can do this by adding three registry keys and a PowerShell command to trigger a DNS lookup for us. We overload the DNS request with sufficient data to generate a meaningful alert for defenders. (We wrap all of this up in one simple .reg file import).
    We discussed some of the advantages and disadvantages of this approach. We are able to do this by leveraging the alert pipeline that you have, perhaps already used.  If you would like to customize these tokens you certainly can too (and you can always run your own custom version of the free Canarytoken server. See the GitHub repo here https://github.com/thinkst/canarytokens).
    Thank you for taking the time to read, and we welcome feedback on ways to improve and refine this new Sensitive command Canarytoken. Thanks again to the Applied Research team at Thinkst for all the testing and feedback to help us bring an idea forward that may prove helpful.


  • Canaries as Network Motion Sensors

    Canaries as Network Motion Sensors


    This post is the first in a series by Canary customers where they detail how they integrated Canaries and Canarytokens into their security practice. This series hopes to showcase how organizations of varying sizes and structures deployed and acted on alerts to improve the security of their networks. Casey Smith recently joined Thinkst Labs, today he’s sharing his experiences with Canaries and Canarytokens from his previous role as a customer.


    Prior to joining Thinkst, I worked for a number of years as the Principal analyst on a security team at an organization of ~3500 people with a highly regulated security practice. Our team was responsible for several systems: Email, Web Application, Proxy Server, Host based EDR, Application Control, Security Analytics, as well as Incident Response, internal testing and penetration testing. I would consider our team a fairly mature security team, with lots of tools, software, and telemetry to inform our security response. We found that Canaries detected activity that our other tooling and telemetry did not.

    Some of these examples have been modified slightly to avoid disclosure of certain internal details.

    Our team prioritized visibility, followed by detection, then prevention. We can’t defend what we cannot see or detect. This philosophy helped our team gain great insight into our network, systems and applications. EDR for example, would allow our team to search for any host, any process, that makes a DNS request. We could then correlate that with other systems to react to unauthorized, or suspicious access. We still found incredible utility from Canaries in these cases and they are a much lower cost than many of the other tools we purchased and deployed. We were able to leverage Canaries to detect both internal and external attacks.

    While no defense is perfect, this model informed our approach, along with a tight feedback cycle between detection and prevention.

    Some questions we constantly tried to ask of ourselves:

    1. How do we know this tool is working?
    2. Have we tested this tool?
    3. How do we know if an attacker is moving around in our network?
    4. What tools work to detect already compromised systems, and lateral movement?

    Below is a diagram of Mandiant’s Standard intrusion model, as attackers traverse from left to right, and I’ve annotated it with where we thought about inserting canaries for detection.

    Once an attacker gains initial access, they often do not know where they are. They have to discover, and enumerate services, targets, find credentials or elevate privileges. In essence they bump around, and this can be a defender advantage. This can be done a number of ways, local and remote:

    1. Port Scanning
    2. Active Directory Enumeration
    3. Internal Sites, SharePoint, Confluence
    4. Network Shares, Documents
    5. Local accounts, Local Privilege escalation
    6. Exploits, Vulnerable services, Misconfigurations (Windows Services, etc.)

    This can be to the defender’s advantage to shape the environment early on for the attacker.  What we want to do is present some tempting targets to the attacker, and have them attempt access. This tipoff should then be enough for us to investigate further, if they trigger an alert.  Tempting targets can be files, (Canarytokens) , or Network Services (Canaries). This blog seeks to share some real-world approaches for creating those targets, as well as challenges and opportunities we faced. Below we explore in detail four scenarios:

    1. SSH Detection on Guest Wireless
    2. Cyber Insurance Documents – CanaryTokens
    3. The Alert that should never happen
    4. Log4J Zero Day Detection, Remediation

    SSH Detection on Guest Wireless

    Like many corporate offices we offered free guest Wi-Fi to visitors. At the time there were few controls and little monitoring there, except for an Acceptable Use Policy. This we decided would be a great place to see if anyone was attempting attacks against our exposed Guest Wi-Fi service. We could then detect attacks and correlate attacks across multiple locations.  We set up a Physical Canary in our Guest Wireless VLAN. It took us about 2 weeks from the time we decided to deploy, to get the necessary approvals and coordination, attested and validated etc… This will vary from organization to organization. The way our Wi-Fi was configured, we put the bird in the first VLAN just past the Access Point. Access Points for example, in our configuration, prevent one Wi-Fi guest from sending packets to another. So the natural choice was to place the Canary in a VLAN that was accessible from any Wi-Fi guest. Finally we had the bird in place. Our team deployed a Linux profile with SSH, Web Server, and Port Scan Detection. We tested the alerting infrastructure with our integrations to ensure the analysts would be ready. Once this was validated, the tripwire was set and we waited. It did not take very long for an alert to fire.

    It took just a few days before we received our first alert on Port Scanning and SSH attempts for the Wi-Fi Canary. We immediately started working our Incident Response process to validate the alert was legitimate. Even if it is a false positive, it has been my experience that teams learn by running the alert to the ground, to help further tune and improve.  Typical alert volumes (even without Canaries) would vary week to week, But on average the team would do a complete investigation on anywhere from 5-10 alerts per week, on the low end. These ranged from malicious email that bypassed our spam filtering, to endpoint alerts on suspicious files, and suspicious URLs visited. In almost every case, even if the alert was a false positive, the investigating analyst learned more about the log sources, and gained experience querying them. Also this allowed us to create documentation and surface any gaps or places where we hit a dead end in an investigation.

    Once we determined the alert was valid, we reached out to the Network team for any indicators, logs or other data on the connected attacking host. Our alert provided us three dimensions, a Time Stamp, a Source IP address, and User Name/Password attempted. We wanted to then research any and all activity for that host as far back as we could get. To our delight, the Network team had immense insight into Wi-Fi. They were able to present us complete logs to validate the alert and also provide details of where the attacking device was in the building!

    For more one example on how see this blog.

    It looked something like this:

    We wrapped up our investigation and dealt with the rogue host that had:

    1. Scanned our Guest Wi-Fi, and
    2. Attempted an unauthorized SSH login.

    This was our first real win with a Canary detection and word began to spread within the organization that we had some pretty exceptional tools. This can be helpful. In reality, it was good testing and team work. Security teams never investigate in isolation, they require close coordination with other teams, web administrators, networking teams, and endpoint teams. This incident helped reinforce our classification of these Canaries as “Network Motion Sensors“. We want to know when someone is attempting to move around on our network. Canaries may extend the range and reach of your detection. Places where you cannot install software on endpoints. Conference Room scheduling panels, IOT/ICS segments, or other sensitive segments etc…

    Cyber Insurance Documents – CanaryTokens

    While reading recent public Ransomware reports, we learned that some Ransomware crews were reading Cyber Insurance Policies and targeting those organizations for payments.

    (See this story for example)

    So we decided that these Policy Documents were a prime candidate for placing Canarytokens. Canarytokens (for customers) allows you to upload internal documents, and tokenize them.

    We embedded a few policies inside a shared folder within the organization. These documents were placed alongside real policies, and located in a read-only global share. It wouldn’t take long before the alert fired and we caught an unauthorized read of these files.

    When we built the alert, it had context on who was authorized within the organization to read these documents. In an attempt to read these, the security team was to follow a rapid escalation route to curate this alert. We had an unauthorized attempt to read these files! We were then able to use the EDR tool to review activity of the user and endpoint that had opened the documents.

    This was a great win for catching unauthorized document access. Some teams will argue that the same events could be fired in with Windows Alerts and logs, and while true, Canarytokens in well placed Word files, provided faster detection and validation.

    Alerts that should never have happen

    Like most companies, we had some segments that were highly protected and we were fairly confident would never be reached. For completeness, we decided to create some fake documentation on an internal web page, then deployed some Executable and DLLs along with fake instructions on usage and access. We hoped this would never go off.

    What happened next was quite unexpected. An internal user with access to the location where we placed the package ran the tokened executable. However, it was WAY out of this user’s role to ever attempt access to this segment. So when teams worry about insider threats, this was a great real-world example of catching those threats. In the end, this particular employee received disciplinary action, due to the exploration and execution of binaries, outside their described role.. Within the Canary portal, or Canarytokens.org, teams can create and upload basic executable files that alert when executed, as well as when the file properties are read. This can immediately alert you to someone attempting to gain access to a more restricted area.

    Log4J Product Zero Day Detection, Remediation

    The final Canary use case I wanted to highlight is related to the Log4J vulnerability. When the Log4J vulnerability (Log4Shell) was announced in early December, our team sprang into action. What you may not have heard, is the private story of how Canaries helped us validate a ZeroDay in a product. At that time we had been working with an Advanced Red team that was struggling to gain access from external only vantage. This team had the ability to create custom exploits for targets unique to their clients. So they had extensive exploit development experience. The Log4J vulnerability timeline seen here.

    When suddenly Log4J was announced, the Red Team reached out privately, Thursday, December 9, late in the afternoon, and informed us we had edge servers vulnerable to this attack, and asked if we could test and validate. This gave our team a head-start in understanding the impact and urgency of this exploit.

    The initial focus was on internal application and services. We hadn’t yet considered that this attack might affect our 3rd Party External infrastructure. At the time, we had some early python scripts to test and validate. Initial compromise and uses of this exploit were of the “exploit and call back variety” . So you would exploit the server, and it would attempt to download a second stage. This was mitigated by our External Firewall rules restricting outbound callbacks. However, we began to see over the next few days that a DNS variety (as depicted in the graphic above) was emerging that could exfiltrate keys or other sensitive host settings over DNS. We were able to verify our DNS logs were accurately recording lookup attempts traceable back to endpoints as well.

    Around this time the canarytokens.org site began to publish free Log4J token strings.

    These were immediately useful for our team to help test and validate any mitigation and controls we had deployed. So we started to create 10-12 Canarytokens, so we could test and review settings. While this may not have been the intended use of those, it really helped our team isolate and contain vulnerable systems, by ensuring we had a safe way to really test the exploit.

    Sample Log4J exploit, sends hostname out over DNS:


    So Canarytokens, for Log4Shell were immensely valuable to our security team, since we could reliably test, nor attempt to use sketch public exploit code, etc…

    Key learnings

    Each of these scenarios helped us learn about how to use Canaries and tokens as part of our security practice. Each of the key learnings are listed below in case they help you with your deployment:

    1. SSH Canary – Integrating alert data into existing tools even across teams can provide more insights than a single alert alone.
    2. Cyber Insurance Documents – Putting the context for what would be unauthorized access in the token comment allowed for immediate identification of malicious behavior versus someone inadvertently opening the wrong file that they were supposed to be accessing.
    3. Alerts that shouldn’t happen – Even for areas where you are pretty sure you have things covered, Canaries and Canarytokens are a quick way to let you know when your assumptions have broken down.
    4. Log4Shell tokens – Tokens don’t have to only be used as tripwires, they can be used as a probing mechanism to understand how your environment really works to secure accordingly.

    Closing Thoughts

    I have written here about four scenarios. I think the operational impact of canaries cannot be understated, for teams with limited budget and support, Canaries and Canarytokens punch well above their weight class. The alerting pipeline and infrastructure as well are incredibly useful. However, it is also important to remember that a Canary alert is never enough to completely convict or evict an adversary. These, in my opinion, are like smoke alarms, or a motion sensor alert. It will take teams working together and ensuring their birds are cared for and tested and ready to go–much like changing the batteries in your home smoke detectors! Teams may want to periodically ensure they have what they need to respond to an alert. Even among all the other tools we had deployed, from Endpoint to WAF, using Canaries helped our team increase the range of our detection capabilities further into the network.

    We hope these examples spark your interest and curiosity into ways organizations are getting value from Canaries and Canarytokens.

    Thank you for reading.

  • Always be Hacking…

    We discussed this Scott Forstall clip internally and figured it was worth sharing since theres so much going on in just 5 minutes.

    Bradley commented on how familiar it felt to how we roll and it’s worth digging in to this little more.

    Quick Background

    In 2007 Apple was not yet a trillion dollar company, but its star was definitely on the rise. Jobs was back, OSX was taking root and the iPod was game changer. (Their market cap was ~$174 Billion).

    Demo Prep.

    They are about to demo the iPhone in a few months and already you notice them sweating the presentation details. Should the CEO of AT&T be on stage? When do we read him in?

    You can watch a zillion tech demos, and you will find demo’ers who look like they practiced on the plane ride in.. instead, this is thoughtful craft.

    Getting AT&T excited

    So they are flying to Vegas to show AT&T the phones.

    Notice again, it’s not just “here’s an invite to our launch – you can have a speaking slot” where some big-shot CEO can waffle about the synergy between the two companies. They want him bought in..

    They are going to demo to blow his socks off because that’s going to make his talk different too..

    (On a much smaller scale) We’ve been using this technique for a while too. For almost every podcast we’ve advertised on, we first requested a private demo with the hosts. You can tell the difference in how they read our ad before and after seeing the demo. You can pay hundreds of thousands of dollars, but you can’t buy that difference in their voice when they’ve seen the win.

    Leaving things to chance

    Now.. the iPhone is clearly revolutionary… In another part of the interview (included in the full clip) Forstall mentions how at that point, it was obvious to them that the iPhone would win. He was using it, and knew it was the future, but with all that confidence they were still leaving nothing to chance.. they are going to demo it and Forstall is planning around it:

    • Is there going to be good cell signal/service in the penthouse or will our demos look bad because of it?

    Again. It’s such dedication to the craft. They have the Jesus-phone. You are talking to a seasoned exec. You could totally hand wave past the bad connectivity.


    They want to make sure it pops out of their bag and works.

    The option for using 4-Seasons Wi-Fi is great and what’s clear again is their dedication to removing the suck from the experience. Anyone would have understood “let me just join the Wi-Fi… -fiddle- these pesky things -fiddle-“ but they don’t. They are working on it before they leave Cupertino.

    I also think it’s awesome that Forstall himself is sweating these details. He’s the head of engineering at a $100 billion dollar company. He isn’t sitting in the suite while IT makes it happen and this isn’t just fake egalitarian kayfabe. It’s a hardwired culture of biasing to action / just making it happen.

    Relentlessly Resourceful

    (It’s subtle, but) Notice also, that they didn’t land the solution because he played his Apple Card. He landed it by acting like an AT&T exec and using their weight. Anyone coulda done this (but very few ppl woulda).

    Paul Graham has a famous term when talking about great startup founders. He calls them “relentlessly resourceful.

    A couple days ago I finally got being a good startup founder down to two words: relentlessly resourceful.

    Till then the best I’d managed was to get the opposite quality down to one: hapless. Most dictionaries say hapless means unlucky. But the dictionaries are not doing a very good job. A team that outplays its opponents but loses because of a bad decision by the referee could be called unlucky, but not hapless. Hapless implies passivity. To be hapless is to be battered by circumstances—to let the world have its way with you, instead of having your way with the world.

    Unfortunately there’s no antonym of hapless, which makes it difficult to tell founders what to aim for. “Don’t be hapless” is not much of rallying cry.

    But finally I’ve figured out how to express this quality directly.

    What would someone who was the opposite of hapless be like? They’d be relentlessly resourceful. Not merely relentless. That’s not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can’t simply plow through them, because you don’t know initially how hard they are; you don’t know whether you’re about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.

    Be relentlessly resourceful.

    What you see from the Forstall conversation (which seems obvious in retrospect) is that this isn’t just a requirement for startups. Relentlessly Resourceful is why Apple pulls off the iPod with a small team. It’s why they prototype the iPhone with a tiny team. It’s why they consistently manage to do what they do. Lots and lots of companies have more engineers than they do, and lots of companies have smart execs… but.. all the money can’t buy you smart execs / engineers who default to action and find their way around walls (instead of just pointing out that the walls exist). That bit is culture.

    Relentlessly Resourceful can almost be written as “always be hacking”. We built Thinkst for this. We don’t just want the hacks to build our company, we built our company so we can do awesomer and awesomer hacks.

    So far, it’s been great for us. It’s leet to see (when you look at Apple) that it scales 🙂

  • Creating REST API Canary endpoints

    Given the importance of REST API endpoints for most networks and applications, we wanted a way to use (existing) Canarytokens, or Canaries to detect unauthorized access to a REST API.

    (Like all things Canary) We wanted something easy to use that delivers immediate value. Here we present several new approaches, and look forward to hearing from the community on the usefulness and ways to increase insight here for network defenders.


    APIs are everywhere and permeate most organization’s daily web based workflows. Both internal and external services often rely on the use of REST APIs. From workstation management to web applications, from complex business logic and application integrations, to payment processing services, APIs form a backbone for all kinds of crucial services. 

    It started us thinking, how we might be able to create and use Canaries and Canaryokens to catch or detect unauthorized REST API endpoint or key usage? Our end goal would be for teams to receive an alert when an API key, or tokened URL endpoint they want to monitor, is used by an attacker.

    (The AWS API-key is already our most popular Canarytoken, with hundreds of thousands deployed world wide, but we wanted to mint API keys for our own internal services and have them be as useful)


    At its core, a REST API call is an HTTP request. What we need is a way to receive a request, log relevant headers or parameters and HTTP methods, then send an alert to the Canary console.

    We can do that.

    There are already a number of HTTP request related primitives built into Canaries and Canarytokens. We can easily leverage these to create a REST API Canary endpoint. This allows defenders to quickly gain insight into unauthorized use of REST APIs within their organization with low setup costs.  

    We will show three types of REST APIs you can consider using:

    1. Using the simple Web Bug Canarytoken.
    2. Using the (fake) Webserver on a Canary.
    3. Creating a Custom Canary TCP Service.

    Setup One : Web Bug Canarytoken 

    This is the most basic form of the request, create a Web / URL Canarytoken, then leave a hint to its use in a document someplace and see if anyone ever runs a script or curl command to trigger the alert.

    We will create a Web Bug Canarytoken, annotate our Token Reminder, for later. We will then use PostMan, an API testing tool to send a sample request to start out. 

    First we will create the Canarytoken, either in our private Canary Console or at canarytokens.org

    Notice the full/returned token-URL is:


    Next, we will modify the generated URL slightly to make it appear more like a REST API endpoint. 

    This ought to do the trick:


    Remember, this URL is just an example. Apart from the hostname and the actual token (in red), you can change all other parts of the token-URL.

    Test with a curl command


    curl http://canarytokens.com/api/v1/users/list/ww0mayx9rro4h2l3keazj0zdk


    Or send requests and track responses with a tool like Postman 

    This generates the expected alert:

    This is a simple example to showcase the captured HTTP request, source IP address and the User-Agent string. We can see the utility of a quick alert or signal that someone finds this url interesting (and by extension that someone had access to the place it was stored).

    However, it is pretty clear by the hostname, that we are hitting a Canarytoken URL. Teams using our commercial Canary offering have the ability to use a custom domain. If you are using the free service, you can abstract away the Domain Name by using a DNS CNAME record.

    We create a CNAME record in a domain we control. We want api.example.com to resolve to canarytokens.com. We add a CNAME record to point api.example.com to canarytokens.com, so that when the request is made, we still receive the alert!


    curl http://api.example.com/ww0mayx9rro4h2l3keazj0zdk/api/v1/endpoint



    We can see that we receive the same alert, even though the hostname was changed. 

    Setup Two: Webserver On Canary

    Canaries have a built-in feature that allows you to configure a Webserver on it with a single click. With another click you can choose one of the default HTTP Skins. This is a great way to create an enticing server that attackers may attempt to interact with. Below is a basic example of applying the Jenkins Login HTTP Page Skin. Enable Webserver settings in the Canary Console:

    When someone browses to the page they will see this.

    (By default this exists to capture the credentials an attacker enters – and lets you know that somebody is messing around where they shouldnt be). A Skin here is simply a set of files, styles, scripts and images that you want to present to a visitor to the page. You can also upload custom skins to create any style you want. But! Even having a basic Canary-website like this allows us to capture REST API request attempts. You can post a reference to the service and usage. This could be in Github or other places with script snippets or command documentation. These are basic examples, but we think you get the idea. Below is a modified example of a REST API that allows you to transfer money between accounts. Place this example near the Accounting or Finance developers and lets see what happens.



    curl http://srv02.example.com/api/v1/transfers \

      -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \

      -d amount=530 \

      -d currency=usd \

      -d destination=acct_acdcD82eZvKYlo2C8675309 \

      -d transfer_group=ORDER_66 \



    When anyone attempts to use this command, we will get the following alert!

    In the above example, we only see the basic attributes of the request. (Canary won’t log the request

    parameters by default). To capture the POST parameters, we add a custom file. Then alert would include

    more context: how much money was attempted to be transferred and to which account!


    For more detail on these parameters see this blog post about custom webroot configurations. In order to capture the POST parameters, we created a Defined Skin, you can read further about this here, Capturing Incident Data on POSTs.

    Setup Three: Canarys Custom TCP Service 

    In addition to the built-in HTTP Page Skins, Canaries allow you to create a Custom TCP Service that listen on a port you define, and gives you control over various interactions. This allows you to mimic services that may reflect websites or a REST API without using the standard HTTP service (80,443). If you would like to learn more about enabling and configuring a Custom TCP Service, be sure to review be sure to review our Service Configuration steps.

    We can expand on the basic examples above.

    We enable the Custom TCP Service on port 8443 in the Canary Console.

    To make sure we don’t easily alert for false positives, we can configure the service to only alert when

    TCP Connections send a string we are expecting by adding it to the Alert String field.


    curl -X POST \

    -H ‘X-Auth-Token:15dd7c486d81899f64643d6618c47a4efeedacdc’ \



    As we can see, our alert was triggered when the X-Auth-Token string in the Header matched our

    configured Alert string. The Alert string is not required, if you are interested in alerting on any

    connection attempts and logging the data sent, then leave the Alert string setting off.

    An additional benefit of a Custom TCP Service is we can create tokens that are context specific.

    The request URI, consists of: {URI-scheme} :// {URI-host} / {resource-path} ? {query-string} .

    Given this is the case, we could create query strings in the request that identify the token.

    curl -G -H  “Authorization: Bearer 15dd7c486d81899f64643d6618c47a4efeedac” \

    http://srv02.example.com:8443/api/v1/resetpwd/subscriptions \

    -d “api- version=2014-04-01-preview” \

    -d “location=ThisisinStevesAzureADAuthenticationscript”

    We never had to create a specific token, we can infer from the Request the key location dynamically. We don’t have to be as obvious in the parameter value either. We simply place these examples in links or in scripts. When the request arrives we receive an alert with the complete HTTP Request.


    One last topic needs to be considered here. How would an attacker find these examples, and what might

    entice them to use them? That is a great question. There are a number of places where details for how to

    connect to a REST API might show up. Much like opening a document with a Canarytoken or a URL

    Canarytoken. We want to find ways that would cause someone to interact with our fake REST API. Here are a few thoughts and ideas. We would love to hear more about what you come up with as well.

    Idea One: Place the REST API Canarytoken example in an Internal code repo, disguised as a  comment,

    or a reply or a code snippet. 

    Idea Two: Place a link in an internal page that calls out to the specific URL, or a file share, or laptop,

    that looks like a developers workspace. There are a number of additional references and resources that

    may also be useful to think about how your organization may find helpful.


    We found the blog below from Detectify to be a helpful resource to get teams started thinking about how

    an attacker might find and use a REST API endpoint or keys they discover.

    Source:  https://labs.detectify.com/2021/08/10/how-to-hack-apis-in-2021/


    We can leverage already existing Canarytokens and Canary HTTP services to receive and alert

    on unauthorized REST API request attempts. With the already existing HTTP primitives teams can extend or embed these examples within their organization and applications to detect and alert on unauthorized use.



    API Examples:







    Hacking APIs ,Breaking Web Application Programming Interfaces – by Corey Ball.

  • A “Safety Net” for AWS Canarytokens

    The AWS API Key Canarytoken (paid and free) is a great way to detect attackers who have compromised your infrastructure. The full details are in a previous blogpost, but in short: 

    1. You go to https://canarytokens.org and generate a set of valid AWS API credentials;
    2. Simply leave those in ~/.aws/config on a machine that’s important to you
    3. Done!

    If that machine is ever breached, the sort of attackers who keep you up at night will look for AWS API credentials, and they will try them. 

    And when they do, we let you know that you’ve been breached.

    When you receive an email/SMS/Slack message letting you know that the AWS API key that you left only on BuildServer-7 in Server-room #12 just got used to login to AWS, you know you have a problem. 

    The underlying Canarytoken infrastructure relies on AWS APIs logging their own execution to CloudTrail. This lets us identify: which IP made the call; which API was executed (including both the service name and function); plus other details about the client executing the call.

    The effectiveness of the AWS API Key Canarytoken shouldn’t be underestimated. Attackers have to try them; they could be the keys to the victim’s kingdom. If they don’t try the keys, they might be missing a golden opportunity. AWS API Keys are about the juiciest bait you can dangle in front of adversaries.

    For defenders, the keys we supply are simple and entirely safe. They’re not tied to anything owned by the defender (the infrastructure sits completely at Thinkst), the keys have no permissions so they can’t be used to do anything, and no agents or software needs to be installed. You simply drop a text file, and wait to be notified if anything happens.

    Between Canarytokens.org and our commercial Canarytokens offering, thousands and thousands of  machines worldwide have AWS API Canarytokens lying in wait for attackers.

    Can AWS Canarytokens be detected?

    This introduces a new goal for the super stealthy attacker. If they find an AWS API key on a server, can they tell if it’s a Canarytoken (or if it is a legitimate key?). Our view is that this only matters if an attacker can determine this without triggering the token. In other words, the Canarytoken fails if attackers can come across API credentials, and perform some test that returns whether or not the credentials are Canarytokens but defenders are never notified of the test. 

    The key (heh) to making this happen as an attacker, would be to find AWS APIs that don’t log their execution, and also reveal information about the calling API Key to the caller.

    For our purposes, there are four classes of error responses from AWS APIs:

    1. Logs to CloudTrail, reveals no information about the API key
    2. Logs to CloudTrail, does reveal account details from the API key
    3. No CloudTrail logs, reveals no information about the API key
    4. No CloudTrail logs, does reveal account details from the API key

    As defenders,  4 is the worst case. If an API fulfills number 4, then Canarytokens can be detected. 3 is not great either, but softens the impact to “attackers can determine where credentials are valid, but not whether they’re Canarytokens”.

    The next bit may surprise those who’ve never worked with it: the AWS API is a mishmash of response codes, error code, error strings, exception names, and data formatting. It is anything but consistent.

    In the past, it’s been the case that a tiny subset of API errors did fulfill the constraints for class 4. The folks at Rhino Security blogged about this (RIP Spencer). While AWS (very slowly) does seem to react to reports on APIs which don’t show up in CloudTrail, that’s not sufficient. As it stands, we’re not aware of class 4 error responses currently in the AWS API, but at the rate at which new APIs are added, that’s little solace for thinking about the future.

    There are certainly class 3 error responses in the current AWS API if you go looking hard enough.

    In summary, the AWS API has a small number of endpoints which don’t log their own usage to CloudTrail, creating a blindspot for defenders. What do we do in those instances?

    IAM Credential Reports

    Scott Piper pointed out that Amazon actually does provide a backup mechanism for identifying credential usage. The IAM service lets you generate and download a report for all your credentials. This is a CSV file where each row belongs to an IAM user, and some of the columns identify when an access key was used, and on which AWS service it was used.

    With the credential report, there are no longer class 3 or 4 error responses in a practical sense; we can also check the credential report to see if an API key was used, so no longer need to just rely on CloudTrail.

    The report has three main drawbacks.

    1. The report can only be generated once every four hours, so at worst there’s a four hour delay between credentials being used, and you seeing the notifications. 
    2. The fidelity of the reports is low; you only can tell the last time the key was used and on which service (e.g. ec2, iam, sts, etc). You can’t tell which function was called. 
    3. You don’t get any client information, including IP addresses or client versions.

    However in spite of these drawbacks, it’s a huge step up in terms of reliability of this token type. Attackers don’t have places to sneakily test their API keys. This safety net means that every key usage will be detected and alerted on, albeit sometimes with lower fidelity artifacts. 

    Even if you didn’t know the attackers IP Address, would it be worth knowing that the key that you placed on \\code-sign-22 was used to login to AWS this morning? Absolutely!

    In fact, if a key is only used on what was previously a class 3 or 4 API, that’s even more of a signal since it implies the attacker is actively trying to avoid detection.

    So we built an AWS safety net, to help us catch attackers who actively try to avoid detection.

    Building an AWS Safety Net

    The architecture is very straight forward. An AWS Lambda function fires on a periodic schedule. It pulls in the credential report from IAM, iterates over all the rows, and determines whether the last recorded use of the key happened more recently than the CloudTrail logs show. If so, the Safety Net kicks in, and sends out the alert:

    Browsing to the history page for the incident, you see an annotation telling you that the Safety Net picked up this usage:

    The Safety Net was deployed to our commercial customers a little while back thanks to Jason, and we’ve recently rolled it out to the free users on Canarytokens.org too.

    Wrapping Up

    AWS API logging previously left a small but significant gap that potentially gave attackers a way to use Canarytoken API keys without triggering alarms. With the deployment of the Safety Net, this gap has been plugged.

  • Canary Rice Toss

    To see a World in a Grain of Sand Rice

    William Blake mh

    If you are on TikTok (or a fan of talk shows) at the moment then, no doubt, your feed has included coloured rice being tossed in the air in the form of song lyrics, beloved cartoon characters, and even famous faces.


    Whilst coloured rice is not a new thing (for most preschool teachers, it is a cheap and effective way to keep kids entertained), a bunch of TikTok-ers have made a living off turning this simple play-thing into a full-on career. And, obviously, when a current trend is well-suited to our logo, we have to give it a go. Here’s how we got there:

    What we used

    • Rice
      • Our whole logo only needed 500g, however, we needed a few attempts to get it right and ended up using about 3kg. White rice is reusable, a jumble of multi-coloured rice, not so much…
    • Food colouring
    • Vinegar
      • 1 teaspoon per cup of rice
    • Large (relatively stiff) rice tossing surface
      • We used a shelf from our cupboard…

    How we did it

    • We started by colouring the rice
      • We mixed 4 cups of rice with 4 teaspoons of vinegar and added food colouring until we got to the colour we wanted. The vinegar helps the colour spread evenly and get absorbed by the rice, so that, when playing with it, the colour doesn’t stain your fingers.

    • Once mixed well (and to your colour-liking), we then dried the rice on trays lined with paper towel. The thinner the layer, the quicker it dries. Ours took about 2 hours.

    • We scaled and sketched our logo onto the board
      • If we had had a projector, this would have been a much quicker + easier process, but alas.
      • Note that the image that you sketch out on the board will be reversed when tossed into the air. We drew the logo on in the correct orientation and then simply flipped the footage in editing
    • Practice makes perfect
      • We had a few attempts with random shapes, just using white rice, so that we could practise the flip motion (and then could reuse the white rice)
      • We found the trick was to not raise the board too aggressively, but drop it as quickly as possible
    • Once we were happy with our flipping technique, we then created our logo using the coloured rice.
      • We packed the rice relatively densely to make sure that, when tossed, the logo came out as clear and vibrant as possible.
    • We got the cameras set up to capture the magic.
      • We used both a GoPro + the SlowMo setting on an iPhone. Whilst both worked, slowing down the GoPro footage in editing gave us better quality.
    • The next step was to toss our Canary logo
      • It took us 3 solid attempts to get our final result (with plenty of sweeping in between)
      • We did not put down a sheet to capture the rice (because…yolo), but we would recommend this method (unless you plan on finding runaway rice in all of your living room crevices for the next few days)



    Easy and cheap way to get creative (and certainly gave us a good laugh). Would recommend.

Site Footer

Authored with 💚 by Thinkst