Adversaries love (ab)using security software – why wouldn’t they?

Security software as a target

Security tooling is widely deployed, typically controllable remotely, and usually runs with elevated privs. This makes attackers greedy for credentials to security back-ends, and we can exploit that greed! In this post we introduce our latest Canarytoken: the CrowdStrike API Key

Adversaries are always on the lookout for credentials, and by placing decoy keys in their path it’s possible to detect them early. 

They love API credentials because they have goals & objectives, and more often than not, API keys provide them with healthy in-roads towards them.. 

Crowdstrike RTR

Security tooling is infamous for being the shadow keys to the kingdom. If we take a look at CrowdStrike’s Real Time Response (a feature of their EDR), it’s a great example of why. RTR is essentially an online remote shell on every endpoint with Crowdstrike installed. With the right access RTR can give an operator the primitives they need to do real work across an environment. 

Attackers know it too, and that’s why those credentials are valuable during hacking operations.

Crowdstrike API Key Canarytoken

Our new Canarytoken lets you generate a valid CrowdStrike client credential in seconds. You then deploy it wherever “helpful creds” tend to accumulate. This could be on security analyst endpoints, in scripts, in config directories(in those folders everyone says they’ll come back to clean).

When an adversary runs into those credentials  they almost have to use them. And when they do, they end up exposing themselves..

How to Create them? Dead Simple:

1. Visit Canarytokens.org and select the Crowstrike API Token;

2. Enter an email address (to receive your notification) and a brief memo to remind you where you are placing this key

3. And that’s it!

Place your token in a single location where attackers will find them and wait.. 

Internals

We create the credential in our own sandboxed tenant (so nothing of yours is ever exposed) and we obtain authentication notifications through CrowdStrikes excellent audit logs (Kudos to CS for being a rare security vendor who makes these easy to consume). 

This is important so we’ll repeat it: there is no risk to your Crowdstrike tenant as everything lives inside ours. (You don’t even have to be running CrowdStrike in your environment, but your attackers won’t know that). Simply copy a few lines of a text file into a single location in your environment, to get the benefit.

Attackers can’t tell the difference, and you obtain that one high fidelity alert, when it matters.

Wrapping up

Give attackers options. Make them pay. The new Crowdstrike API Key Canarytoken is live and ready for you to try out and deploy.

Generate one here: https://canarytokens.org/

References:

https://attack.mitre.org/techniques/T1219

https://securelist.com/miner-campaign-misuses-open-source-siem-agent/114022

https://cloud.google.com/blog/topics/threat-intelligence/unc3944-targets-saas-applications

Last week we concluded the deal to acquire 100% of UK-based DeceptIQ. We welcome them to the flock.

DeceptIQ is built by red-teamers with a deep desire to turn the tables on attackers. In our ten years of doing Canary, we’ve never seen such a strong natural alignment.

We are super excited to write future chapters (to help defenders win) together.

Introduction

Canarytokens have proved themselves over the last decade as an easy-to-deploy breach detection tool. Our free Canarytokens service has supported AWS API keys since 2017. The concept is straightforward: you sprinkle decoy API keys in your code repos / Lambda configurations / virtual machine disks; when the credentials are used by attackers, you’ll get an alert in your mailbox. They make an excellent (and simple) way to identify malicious actors inside your infrastructure, in the early stages of the attack when attackers are gathering information about your environment.¹

Empirically, they rock! Grafana recently posted how Canarytokens were the mechanism that caught their breach.

This week we’re excited to broaden our support for decoy AWS assets by introducing the AWS Infrastructure Canarytoken, available immediately on https://canarytokens.org without signup (for free!). The AWS Infrastructure Canarytoken is quite simple: we help you deploy decoy resources in your AWS account. These assets will look enticing to attackers; when your AWS account is compromised and they explore your account, they’ll see the decoys, interact with them, and you’ll get an alert. Previously you could create decoy resources and alerting rules manually; with our new addition the process is much quicker and more straightforward.

The way it works is that we help you design a Terraform module that you apply to a specific region of your choosing in your AWS account. The module will set up decoy assets (DynamoDB tables, S3 buckets, SSM Parameters, SecretsManager Secrets, and SQS Queues) in your AWS account’s region, and configure your Cloudtrail logging such that events related to the decoys will generate alerts delivered to your mailbox via Canarytokens.org.

In this post, we’ll walk through how to create and deploy an AWS Infrastructure Canarytoken, and describe how we address the security risks associated with cross-account data flows.

Pre-requisites

To create a new AWS Infrastructure Canarytoken in an AWS account you’ll need:

• A working AWS account (let’s call it 123456789012 for our examples below) with a region where the decoys will be created (we’ll go with the venerable us-east-1 in our examples below).
• An AWS CLI session with permission to create a new IAM Role, IAM Policy, and attach the policy to the role, all in your AWS account. I.e. $ aws sts get-caller-identity works, and is empowered to make IAM changes. The easiest way to get a session is via CloudShell from your AWS console.
• Terraform is installed, and set up to add resources in the AWS account. You might have added credentials as environment variables, or inside Terraform files, the provider has several options.
• Your target AWS region is already enabled. If you’ve recently enabled the region, it can take up to several hours before it is fully enabled.

Creating an AWS Infrastructure Canarytoken

There are three steps to creating the AWS Infrastructure Canarytoken. First, you’ll set up the AWS integration with Canarytokens.org. Then you’ll create a decoy infrastructure design. Finally, you’ll deploy it with Terraform.

Step 1: Setting up the AWS integration

1. Browse to https://canarytokens.org, and click on the “AWS Infra” panel:

2. Under the “Canarytoken Settings”, enter the AWS account number and select the region (the region must already be enabled in your account, review your list here):

3. For the alert details, enter an email address where we’ll send the alert, and a reminder message we’ll include in the alert. We recommend a description of the AWS account, so you quickly can tell which account was affected:

4. Click “Start creating Canarytoken”.
5. The next screen shows three AWS CLI commands, which you need to run in the authenticated AW CLI session. Click “Copy AWS CLI snippet”, and paste into your shell:

   

   These commands will create a new IAM Policy, a new IAM Role, and attach the role to the policy. This will allow Canarytokens.org to fetch the names of certain asset types in your account, in order to generate decoys with similar names. The policy contains the asset types we list, and is a least-privilege policy. For example, it allows us to list your current S3 bucket names, but not read anything inside the buckets (including object names). Ditto for DynamoDB tables, SSM Parameters, SecretsManager secrets, and SQS Queues.
6. When you’ve created the Policy, Role, and attachment, click “Continue”.
7. Canarytokens.org will connect to your AWS account, and perform a name-only inventory of your S3 buckets, DynamoDB tables, SSM Parameters, SecretsManager secrets, and SQS Queues. It will then generate a proposed decoy design for you (see below for more details on how we do that), and show you an editor to make changes to the design.

Step 2: Creating a decoy infrastructure design

1. In the design editor, click on the decoy types to review the design. If you’d like to add additional decoys, simply click the “+ … Decoy” button:

2. If you’d like to edit the name, or remove this decoy, use the “Edit” or trashcan icons respectively:

3. We use Google Gemini to assist with generating names; each Canarytoken gets up to 50 name generation requests and we show your status as you build your design. Note that generating the first plan consumes part of your quota, so you’ll always see less than 50 on the initial page load:

   

   When the count drops to zero, then we’ll continue to simply generate random names, no longer relying on Gemini.
4. When you’re happy with your design, click “Save configuration”.

Step 3: Deploying the design with Terraform

On the final step, we save your design into a Terraform module which we host for you. What remains is for you to add a reference to the module in your Terraform file, and apply the plan:

1. Click the copy icon:

2. In your Terraform plan, include the module, and save:

3. Run $ terraform init to install the module.
4. Run $ terraform apply to deploy the decoy infrastructure.
5. Your decoy assets are now in place.
6. You can test by browsing one of the decoys in the AWS Console, or examining it through the AWS CLI. For instance, if one of your decoys was an S3 Bucket with an object inside, you could list it to trigger the Canarytoken like this: $ aws s3 ls s3://devproject/service-creds

With that, your first AWS Infrastructure Canarytoken deployment is done! Let’s take a closer look at what’s happening behind the scenes.

How we generate decoy names

When building this new Canarytoken type, we wanted to make the decoys blend in with your current AWS infrastructure.² This feature stands on two legs: an inventory of your AWS account, and a name generator.

The inventory process sets up a temporary read-only connection between Canarytokens.org and your AWS account, which allows us to list the names of specific asset classes in the region you picked (except for S3 buckets, which are globally unique). 

Once we have your current asset names, we’ll pass them to Google Gemini to propose additional names that are similar to yours. We also base the number of proposed decoys on your current count. There is a cost associated with each name generation request made to Gemini. We don’t pass it on to you, which means we have to watch our spend. We solve this through assigning a quota to each Canarytoken for name generations.

The proposed decoy names will match your current assets, along with additional enticing names.

Cross-account access

Separately from name generation, we’ve adopted an architecture in which we isolate functions in specific AWS accounts, to limit risk in any single account. To this end, we have to carefully consider how your AWS accounts can send us data.

There are two directions for access. When creating the Canarytoken, you set up temporary read-only access to allow Canarytokens.org to pull information from your account. This access is short-term and only available during the creation or edit stages. We use AWS’ recommended pattern for access to third party accounts; it hinges on a so-called ExternalId. (which is part of the IAM policy you create). When you’ve created your Terraform module at Canarytokens.org, we delete the ExternalId so we don’t retain long-term inventory access. You could remove the IAM policy and IAM role yourself if you like, but we still wanted to proactively sever the connection from our side, and using an ExternalId does just that.

The second direction of access is long-term, and involves your AWS account pushing select Cloudtrail events to Canarytokens.org. This is how we monitor the decoys for access. We take care to set up Cloudtrail such that only events related to the decoys are sent to us. An EventBridge rule in your account filters events, before sending them on to us. The filter is an allowlist; we’ve taken care to specifically limit it to known events about the decoys, ensuring that we won’t accidentally see events related to non-decoys. Naturally this is inspectable by looking at the Terraform module.

Costs

Deploying this Canarytoken into your account will consume AWS resources and count towards your quotas or billing. This will typically be single-digit dollars per month. The highest fixed cost is SecretsManager Secrets, which AWS charges $0.40 per secret per month. Apart from that, all others are API call-based pricing. Decoy resources aren’t hit regularly by design, so their charges will typically be cents per month.

For obvious reasons, your AWS bill is for your account. We pay for our usage of AWS to process your incoming EventBridge events when they hit our account, and we pay for Gemini.

Editing the decoys at a later stage

We’ve also included the ability to later edit your Terraform plan. By accessing the Manage link for your Canarytoken, you can re-enter the editor. Note that you’ll need to tell us the ExternalId so that we can re-inventory the account. The wizard describes this in more detail.

Final thoughts

We’ve been noodling on this for a bit, as an extension of the previous work we’ve done, and we’re excited to release it. Dev, Lisa, and Vittoria have created a powerful new Canarytoken type. Like our other Canaries and Canarytokens, they deploy quickly and give you the heads-up when you need it most. We’d love it if you took it for a spin!

Footnotes

1. On the commercial side, we’ve also supported S3 Bucket Canarytokens for a long time, in addition to the AWS API Keys. ↩︎
2. We don’t believe this is a requirement for Canarytokens generally, however it seemed like a fun challenge. ↩︎

You’re moments away from finishing a feature you’ve been working on for the last two weeks when you get a Slack notification that the frontend test pipeline has failed for the 824th time that year. 

It’s the same handful of flaky tests that fail whenever there’s a half-moon.

You make a note to fix these tests and get back to finishing that feature.

We were in this situation and asked ourselves whether we enjoyed building and maintaining our frontend test system. The answer was no, so we tore it down and built something we could be proud of.

Getting your frontend testing infrastructure stable is tough. Timing is tricky to get right when your tests are at the whim of network requests and browser rendering cycles. However, with the right tools and a solid foundation, you can do it, and it’s worth the effort.

A promising testing pipeline isn’t just about catching issues; it’s a force multiplier for your development team, empowering them to move faster, confidently, and focus on doing their best work.

In this post, we walk through why we switched from Cypress to Playwright, how we made the switch, and what the outcomes have been.

What was wrong with our old testing system anyways?

Our Canary Console frontend has decades of person-years poured into it. We sweat the small details. A while back, we settled on Cypress for running frontend integration tests, but over time, it’s become that mystery Tupperware in the office fridge — nobody wants to open it because of what you might find, but it’s still there, singing its siren song.

Our integration tests need to interact with live Consoles, and each Console has various settings and features that can be enabled or disabled (for example, we can turn SAML authentication on or off for each customer, which affects part of the frontend). We were hitting limitations in Cypress and the testbed architecture, which took too much time to work around.

Here’s an outline of how frontend testing was integrated into our CI flow:

1. The code landed on our main branch.
2. A Github Action would start a Cypress agent.
3. The Cypress agent interacted with a single, persistent, live Canary Console.
4. The Action would send failures to Slack.

In this design, we never reset the Console state between testing rounds. If tests don’t clean up after themselves, the next round of tests will run against an unknown state.

Since the tests shared a single Console, we couldn’t run them in parallel, so either developers were bottlenecked on the single testbed, or they tested on non-standard testbeds.

It was a mess; so much that no one wanted to write tests, and our frontend team didn’t want to be involved with the project. So, how did we get folks excited about writing end-to-end tests?

Yearning for the ocean

If this system was perfect, what would that look like? Why not aim at that? We came up with:

1. The tests would run on every commit, even on feature branches.
2. The test suites would have zero flaky tests. Why can’t the tests run for months without failing due to flakiness?
3. The test suite would execute rapidly. Prompt feedback is essential.
4. The test system would be portable. It would run in our CI and on our developers’ local machines.
5. Writing tests should be enjoyable; at least, we shouldn’t want to bang our heads on our desks while writing them. If the tests aren’t easy to write, how will you craft long, comprehensive test cases?

Having these points as our guiding constellation, we knew that getting close would mean raising our team’s floor to a new level. The plan became:

1. Make a repeatable test environment in containers.
2. Move from Cypress to Playwright for writing and running tests.
3. Focus on flaky tests.
4. Ensure our main UX flows were covered.

Foundation

We moved our entire stack into a collection of Docker containers we could spin up anywhere and anytime.

We could run the container system with a fixed pristine database that resets each time the stack starts up, meaning our state is always the same. The only thing that would change would be the code. Using Docker was a great start at solving our stability issue.

Our setup solved 1 and 4 of our dream list, and gave us a headstart on 2.

For #1, Github Actions can run arbitrary Docker Compose setups. We could spin up our entire stack for each commit and run the new code against that isolated environment.

For # 2, The stability afforded us by being able to reset our state when we spun up the containers meant that we had a good base for a system that wasn’t flaky, thanks to our pristine database that never changes. There was still work to fix individual tests, which we’ll explore in more detail later.

For #4, Since portable environments are what Docker is all about, the testing environment could run in GH Actions and on our local machines. Both testing environments are identical.

The foundation set us up for success, but we needed to ensure that our test cases followed suit.

Joy for devs and force multiplier (why we moved to Playwright)

As trite as it sounds, writing Cypress tests didn’t spark joy. There is a list of areas where Cypress pales compared to Playwright (and we’ll get into those shortly), but we didn’t like using the tool at a base level.

We decided to trash all our Cypress tests and start again in Playwright. If you have a collection of Cypress tests that work well, keep those around. We didn’t have that many, and almost none of those tests were useful. After we weighed up the pros and cons, switching was a no-brainer.

Cypress Eccentricities

The Cypress test framework has many idiosyncrasies that add unneeded friction to writing and reading tests.

Cypress commands aren’t promises. They aren’t synchronous, either. Commands are added to a queue and only run after the function returns. Take a look at this example:

const username = cy.get('#username').invoke('val');
console.log(username); // This logs a "Chainable" object, not the actual value

Logging the username variable returns a Cypress chainable, not a resolved DOM element or value. When console.log() runs, Cypress has yet to execute the cy.get command. To get the value, you need to use a .then(), which waits for the command to resolve.

Consider a test that verifies whether the sum of a shopping cart’s item prices, VAT, and shipping cost matches the total displayed.

In Cypress, there are two ways to do this. You either need to use then, which waits for your previous command to complete and then executes your callback:

describe('Cart total exact match validation', () => {
  it('should exactly equal item price + shipping + VAT of $10.00', () => {
    cy.visit('/cart');

    cy.get('[data-testid="cart-item"] [data-testid="item-price"]').invoke('text').then((itemPriceText) => {
      const itemPrice = parseFloat(itemPriceText.replace('$', ''));

      cy.get('[data-testid="shipping-cost"]').invoke('text').then((shippingText) => {
        const shippingCost = parseFloat(shippingText.replace('$', ''));

        cy.get('[data-testid="vat-amount"]').invoke('text').then((vatText) => {
          const vatAmount = parseFloat(vatText.replace('$', ''));

          const actualTotal = itemPrice + shippingCost + vatAmount;
          const expectedTotal = 10.00;

          expect(actualTotal).to.equal(expectedTotal);
        });
      });
    });
  });
});

Or you can use aliases, which are essentially variables you can reference in other parts of your Cypress test:

    describe('Cart total exact match validation with aliases', () => {
      it('should exactly equal item price + shipping + VAT of $10.00', () => {
        cy.visit('/cart');

        // Get item price and alias it
        cy.get('[data-testid="cart-item"] [data-testid="item-price"]')
          .invoke('text')
          .then(text => parseFloat(text.replace('$', '')))
          .as('itemPrice');

        // Get shipping cost and alias it
        cy.get('[data-testid="shipping-cost"]')
          .invoke('text')
          .then(text => parseFloat(text.replace('$', '')))
          .as('shippingCost');

        // Get VAT and alias it
        cy.get('[data-testid="vat-amount"]')
          .invoke('text')
          .then(text => parseFloat(text.replace('$', '')))
          .as('vatAmount');

        // Use cy.then to access all aliases
        cy.then(function () {
          const actualTotal = this.itemPrice + this.shippingCost + this.vatAmount;
          const expectedTotal = 10.00;
          expect(actualTotal).to.equal(expectedTotal);
        });
      });
    });
  

Playwright Sensibilities

There is only one style in Playwright; it doesn’t contain any framework-specific knowledge. If you’re familiar with writing JavaScript, you’ll understand it quickly.

Native JavaScript promises form the foundation of Playwright, which allows us to wait for the result of any async operation and store it in a variable. This style of programming aligns with how we write code daily:

    test('Cart total should exactly equal item price + shipping + VAT of $10.00', async ({ page }) => {
      await page.goto('/cart');

      const itemPriceText = await page.locator('[data-testid="cart-item"] [data-testid="item-price"]').textContent();
      const itemPrice = parseFloat(itemPriceText?.replace('$', '') || '0');

      const shippingText = await page.locator('[data-testid="shipping-cost"]').textContent();
      const shippingCost = parseFloat(shippingText?.replace('$', '') || '0');

      const vatText = await page.locator('[data-testid="vat-amount"]').textContent();
      const vatAmount = parseFloat(vatText?.replace('$', '') || '0');

      const actualTotal = itemPrice + shippingCost + vatAmount;
      const expectedTotal = 10.00;

      expect(actualTotal).toBe(expectedTotal);
    });
  

As developers, we want the ideas in our heads to flow freely into our IDEs (or terminals, if you’re a psychopath), with as little friction as possible. Framework-specific knowledge comes at a cost. If you aren’t writing Cypress tests every day, you’ll need to refresh your memory each time you need to write a test. Continuity between your tests and code means moving between the two is effortless.

Developer Experience

Playwright has great support inside VS Code. Want to step through your test code with breakpoints? You’ve got it. You just fixed a test and want to test it to see if it works. Rerun it directly from inside your IDE with a click of a button. It’s smooth, easy, and a treat to use.

Speed

Playwright’s performance was the feature that initially caught our attention. We planned to stay with Cypress and rework our foundation (moving our stack into Docker containers). Out of interest, we looked at which testing framework the industry used, and there was a consensus that Playwright was the new “it girl”.

We started with our longest-running Cypress test (creating one of each of our Canarytokens), which took around 5 minutes to execute with Cypress using Chrome only.

Migrating the identical test to Playwright, it now takes 1 minute and 30 seconds to complete, running against Chrome, Firefox, and Webkit. For a developer, few things are as gratifying as watching performance numbers drop—it’s like pressure-washing a grime-caked driveway.

How is Playwright so much faster? Playwright runs each test in a separate browser context (essentially an incognito window). It interacts with the browser contexts by means of the DevTools protocol for Chromium and custom protocols for Firefox and Webkit. Because tests are isolated, Playwright can run multiple tests simultaneously.  Cypress runs tests in-process, meaning the tests run inside the browser in the same execution context as the app you are testing.  Cypress can only run a single test simultaneously; parallelism requires using Cypress Cloud.

All adds up

Ensuring your team doesn’t wallow in Cypress code, trying to remember framework-specific gotchas, or waiting three hours for the test suite to finish means your team can focus on writing useful test cases.

One of our team members pointed out that our CI test report artifact was 700mb, and downloading it from GH took 30 minutes. We could have shrugged it off, but we took half an hour to investigate whether we could change the tracing to only include failed test data. We could. It’s a tiny thing, but had we left it, we might never have gotten around to it, and a 30-minute wait would have become the norm.

Concise code makes it easier to understand a test’s flow. Respecting your developers’ time means they’ll have more time to do the important stuff. Having a system people enjoy makes pushing through tough patches easier.

Fight the Flake and knowing your systems

It’s easy to ignore that one flaky test. “It only fails every other time, it’s fine.” Eventually, as you add more and more tests, you’ll find yourself in a situation with 20 flaky tests. Twenty flaky tests distracting your developers every time they fail.

We set a zero-flaky-test policy. Whenever we completed our tests, we’d have to run the suite with –repeat-each=50, which reran the new tests 50 times. If they failed once, we’d need to figure out why and fix it.

Running the tests repeatedly might seem harsh, but it meant that these tests would be as solid as a Nokia 3310. It also forced us to get to know Playwright, our testing infrastructure, and our UI. 

Initially, running tests concurrently would lead to loads of inconsistent failures. As the frontend team, we could have left this and chalked it up to flakiness, but we had a no-flakiness policy. We had to figure out why this was happening. Debug web server startup scripts aren’t built for concurrency. Our production Consoles use Nginx and uWSGI to facilitate our frontend communication with our backend.  So, we aligned our test setup with our production services. No more flaky test failures; as a bonus, our tests ran even faster.

This increase in speed also allowed us to find a race condition that occurred while deleting a Flock, which would break the UI on page load. This bug had been present for years. We could have ignored it but that damn “no flakiness” policy! *shakes fist*

While building our user management test suite, we noticed the tests took longer and longer to run each time they were repeated (--repeat-each=50). Every time we deleted a user, every other admin user would get an email mentioning that somebody had removed a user. This growing list of emails would start interfering with the test timings, and tests would fail. The compounding delay could mean a user might wait 10 seconds for an API request to return. Also a bug that was present for a while, and we wouldn’t have found this if we had ignored it.

These things might seem unimportant for standard test runs (that don’t repeat), but as you add more tests, the flakiness will start to add up. Squash the bugs when you find them so you can return to the important stuff.

Make sure you’re testing the important stuff

Our transition between Cypress and Playwright wasn’t just a straight port. We also wanted to write new test cases that completely covered core feature functionality. We started with the five most important features our customers use.

For us, we’re all about letting customers Know. When it matters. For the tests, this meant focusing on alert management and testing our most popular Canary configuration flows to ensure folks could get those alerts. Alerting and setting up Canaries are the most critical aspects of our interface. Customers spend their time in these areas, and we want them to be bug-free.

In our Cypress test suite, we had little to no focus on what was or wasn’t an important test. For example, we had a test to assert that the text on our 404 page was correct. Now, by itself, that test is okay, but why would you write a test for that when the Cypress suite has no alert management tests, our company’s primary focus?

We set a plan by focusing on the critical stuff and ruthlessly avoided busy work.

What’s a good test?

A great test behaves as a user would. It should mimic the actions a real person performs. Making sure the elements are visible isn’t enough. For example, suppose the feature is a configuration wizard to set up a device. In that case, the test should step through the entire wizard, fill in forms and tick checkboxes, and then complete the wizard with an assertion to confirm the device settings have changed. The tests should cover all permutations of the wizard, looking for bugs in every nook and cranny.

Paul wrote a suite of tests for our Windows File Share service, one of our most popular Canary services.

Users can tailor their file share through the UI by adding, removing, renaming, or uploading custom files. An optional feature also generates a file structure automatically. Here’s an overview of the feature from a recent webinar:

There’s a lot of moving parts. Here’s what we tested:

1. Turning the service on and off
2. Validate user input
3. Turning System Shares on and off
4. Turning alerting on and off on Authentication attempts
5. Enabling and disabling of a File Share Instance
6. Adding a new File Share Instance
7. Validate user input for a File Share Instance
8. Creating a new folder in the file tree
9. Editing a folder name
10. Deleting a folder
11. Adding all of our [7] basic file types (.docx, pdf files)
12. Turning on and off Industry Specific Shares (our feature that generates the share based on your industry)
13. Selecting an industry generates a new share
14. Adding miscellaneous departments to the new share
15. Regenerating the Industry Specific Share
16. Custom upload file size limit validation

These tests ensure that our Windows File Share works as expected, even when we’re not around. It’s important to note that we’re testing features as if we were a user. We’re testing actual functionality and not just ensuring the word “Windows File Share” appears in the DOM. Anyone making changes to this feature in the future can rest assured that we’ve got their back.

Writing tests like this is hard work, and ensuring they aren’t flaky is also hard. The thing with writing good tests upfront is that they become the blueprint for all the following tests. Solid tests provide value every time your CI runs. They hold the shape of your product and ensure you aren’t letting your customers down.

How Close did we get to our ideal test system

In the end, we got pretty close.  We’ve had 20 failure alerts in 148 commits (around 2 months of work) to main.

A handful of these were due to an upstream bug, which we’ve worked around by installing the latest Docker Compose.

A few of them were due to legitimate bugs we introduced; whoop, whoop!

Some were due to flaky tests that managed to sneak past our --repeat-each=50 rule. We put these tests on pause while we looked at their implementation. We needed to overhaul a few of these tests before re-enabling them. We decided to leave the more stubborn tests disabled because we concluded that the value they added wasn’t worth the effort. Remember, only focus on the important stuff! Low-value tests that distract your developers aren’t worth it!

As of writing this, we wrote 311 rock solid test cases across our main feature set:

1. Alerting and Alert Management
2. Canary Configuration
3. Canarytoken Configuration
4. User management
5. Authentication

We’ve got an excellent foundation for building even better tooling in the future.

Tear Down

One of our core principals at Thinkst is that we are a refuge from mediocrity. It’s difficult to do something well, it takes work. We put in the work. If something isn’t where we want it to be, we iterate until it gets there.

We have always known that there was coolness to find in fixing our test pipeline, but we were always busy doing more important work.

If someone had asked me a few months ago whether our Windows File Share feature worked, I wouldn’t have known unless I manually checked. Now, I know it works. If it didn’t, our test suite would have notified me.

Make time for your team to explore things like this. You’ll find a lot of great stuff here, and it might be a force multiplier for your team. You might build something you can’t imagine doing without.

At Thinkst, we build tools to make attackers’ lives harder and defenders’ lives easier. Our latest Canarytoken does exactly that—introducing the SAML IdP App Canarytoken (already available on canarytokens.org, but now available on customer Consoles too!)

Where our Fake App Canarytokens for iOS and Android detect badness at the device level, SAML IdP App Canarytokens help at the identity level. Organisations rely on Single Sign-On (SSO) to manage authentication across their cloud applications. Attackers know this and target identity providers (IdPs) as a high-value entry point into enterprise environments.

By setting up a fake SSO app in your IdP dashboard, you create a high-fidelity tripwire. If an attacker stumbles upon the app and attempts to access it, you immediately receive an alert identifying the compromised account. Early warning of identity compromise helps you react before an attacker can escalate privileges or move laterally within your environment.

Creating a SAML IdP App Canarytoken

Create a Canarytoken (either on your Console or canarytokens.org) by choosing ‘SAML IdP App’ from the Canarytokens list.

Select an app to impersonate from the dropdown. Leave a reasonable comment to remind yourself where you will deploy the Canarytoken (e.g. ‘Fake Salesforce app on Okta’). If you want the app to redirect to a specific URL, enter it in the ‘Send the user to this URL on login (Optional)’ box.

Tap the ‘Create Canarytoken’ button. Download the app icon to use on your dashboard, and use the ACS URL and Entity ID to create a SAML app in your IdP.

Assign it to users and/or make it discoverable. Your token is now set up, and will blend in seamlessly with other apps in your organisation!

Validation

Unique to the Console version of this token is the ability to enable SAML request validation by uploading the SAML metadata file generated by your IdP after creating the App. With it we can cryptographically validate incoming SAML requests and only alert on legitimate login attempts, adding an extra layer of fidelity to the signal that a user on your IdP has been compromised.

Conclusion

Attackers thrive in silence—Canarytokens break that silence when it matters. In the age of cloud platforms and SSO dashboards, the attack surface is distributed, and a robust defense requires spotting attackers where they are. This Canarytoken catches a very specific case (identity compromise), and we think it can form a valuable part of defenders’ arsenal.