We discussed this Scott Forstall clip internally and figured it was worth sharing since theres so much going on in just 5 minutes.

Bradley commented on how familiar it felt to how we roll and it’s worth digging in to this little more.

Quick Background

In 2007 Apple was not yet a trillion dollar company, but its star was definitely on the rise. Jobs was back, OSX was taking root and the iPod was game changer. (Their market cap was ~$174 Billion).

Demo Prep.

They are about to demo the iPhone in a few months and already you notice them sweating the presentation details. Should the CEO of AT&T be on stage? When do we read him in?

You can watch a zillion tech demos, and you will find demo’ers who look like they practiced on the plane ride in.. instead, this is thoughtful craft.

Getting AT&T excited

So they are flying to Vegas to show AT&T the phones.

Notice again, it’s not just “here’s an invite to our launch – you can have a speaking slot” where some big-shot CEO can waffle about the synergy between the two companies. They want him bought in..

They are going to demo to blow his socks off because that’s going to make his talk different too..

(On a much smaller scale) We’ve been using this technique for a while too. For almost every podcast we’ve advertised on, we first requested a private demo with the hosts. You can tell the difference in how they read our ad before and after seeing the demo. You can pay hundreds of thousands of dollars, but you can’t buy that difference in their voice when they’ve seen the win.

Leaving things to chance

Now.. the iPhone is clearly revolutionary… In another part of the interview (included in the full clip) Forstall mentions how at that point, it was obvious to them that the iPhone would win. He was using it, and knew it was the future, but with all that confidence they were still leaving nothing to chance.. they are going to demo it and Forstall is planning around it:

• Is there going to be good cell signal/service in the penthouse or will our demos look bad because of it?

Again. It’s such dedication to the craft. They have the Jesus-phone. You are talking to a seasoned exec. You could totally hand wave past the bad connectivity.

But.They.Don’t.

They want to make sure it pops out of their bag and works.

The option for using 4-Seasons Wi-Fi is great and what’s clear again is their dedication to removing the suck from the experience. Anyone would have understood “let me just join the Wi-Fi… -fiddle- these pesky things -fiddle-“ but they don’t. They are working on it before they leave Cupertino.

I also think it’s awesome that Forstall himself is sweating these details. He’s the head of engineering at a $100 billion dollar company. He isn’t sitting in the suite while IT makes it happen and this isn’t just fake egalitarian kayfabe. It’s a hardwired culture of biasing to action / just making it happen.

Relentlessly Resourceful

(It’s subtle, but) Notice also, that they didn’t land the solution because he played his Apple Card. He landed it by acting like an AT&T exec and using their weight. Anyone coulda done this (but very few ppl woulda).

Paul Graham has a famous term when talking about great startup founders. He calls them “relentlessly resourceful.”

A couple days ago I finally got being a good startup founder down to two words: relentlessly resourceful.

Till then the best I’d managed was to get the opposite quality down to one: hapless. Most dictionaries say hapless means unlucky. But the dictionaries are not doing a very good job. A team that outplays its opponents but loses because of a bad decision by the referee could be called unlucky, but not hapless. Hapless implies passivity. To be hapless is to be battered by circumstances—to let the world have its way with you, instead of having your way with the world.

Unfortunately there’s no antonym of hapless, which makes it difficult to tell founders what to aim for. “Don’t be hapless” is not much of rallying cry.

…

But finally I’ve figured out how to express this quality directly.

…

What would someone who was the opposite of hapless be like? They’d be relentlessly resourceful. Not merely relentless. That’s not enough to make things go your way except in a few mostly uninteresting domains. In any interesting domain, the difficulties will be novel. Which means you can’t simply plow through them, because you don’t know initially how hard they are; you don’t know whether you’re about to plow through a block of foam or granite. So you have to be resourceful. You have to keep trying new things.

Be relentlessly resourceful.

What you see from the Forstall conversation (which seems obvious in retrospect) is that this isn’t just a requirement for startups. Relentlessly Resourceful is why Apple pulls off the iPod with a small team. It’s why they prototype the iPhone with a tiny team. It’s why they consistently manage to do what they do. Lots and lots of companies have more engineers than they do, and lots of companies have smart execs… but.. all the money can’t buy you smart execs / engineers who default to action and find their way around walls (instead of just pointing out that the walls exist). That bit is culture.

Relentlessly Resourceful can almost be written as “always be hacking”. We built Thinkst for this. We don’t just want the hacks to build our company, we built our company so we can do awesomer and awesomer hacks.

So far, it’s been great for us. It’s leet to see (when you look at Apple) that it scales 🙂

Given the importance of REST API endpoints for most networks and applications, we wanted a way to use (existing) Canarytokens, or Canaries to detect unauthorized access to a REST API.

(Like all things Canary) We wanted something easy to use that delivers immediate value. Here we present several new approaches, and look forward to hearing from the community on the usefulness and ways to increase insight here for network defenders.

Challenge: 

APIs are everywhere and permeate most organization’s daily web based workflows. Both internal and external services often rely on the use of REST APIs. From workstation management to web applications, from complex business logic and application integrations, to payment processing services, APIs form a backbone for all kinds of crucial services. 

It started us thinking, how we might be able to create and use Canaries and Canaryokens to catch or detect unauthorized REST API endpoint or key usage? Our end goal would be for teams to receive an alert when an API key, or tokened URL endpoint they want to monitor, is used by an attacker.

(The AWS API-key is already our most popular Canarytoken, with hundreds of thousands deployed world wide, but we wanted to mint API keys for our own internal services and have them be as useful)

Solution:

At its core, a REST API call is an HTTP request. What we need is a way to receive a request, log relevant headers or parameters and HTTP methods, then send an alert to the Canary console.

We can do that.

There are already a number of HTTP request related primitives built into Canaries and Canarytokens. We can easily leverage these to create a REST API Canary endpoint. This allows defenders to quickly gain insight into unauthorized use of REST APIs within their organization with low setup costs.  

We will show three types of REST APIs you can consider using:

1. Using the simple Web Bug Canarytoken.
2. Using the (fake) Webserver on a Canary.
3. Creating a Custom Canary TCP Service.

Setup One : Web Bug Canarytoken 

This is the most basic form of the request, create a Web / URL Canarytoken, then leave a hint to its use in a document someplace and see if anyone ever runs a script or curl command to trigger the alert.

We will create a Web Bug Canarytoken, annotate our Token Reminder, for later. We will then use PostMan, an API testing tool to send a sample request to start out. 

First we will create the Canarytoken, either in our private Canary Console or at canarytokens.org. 

Notice the full/returned token-URL is:

http://canarytokens.com/traffic/ww0mayx9rro4h2l3keazj0zdk/contact.php

Next, we will modify the generated URL slightly to make it appear more like a REST API endpoint. 

This ought to do the trick:

http://canarytokens.com/api/v1/users/list/ww0mayx9rro4h2l3keazj0zdk

Remember, this URL is just an example. Apart from the hostname and the actual token (in red), you can change all other parts of the token-URL.

Test with a curl command

curl http://canarytokens.com/api/v1/users/list/ww0mayx9rro4h2l3keazj0zdk

Or send requests and track responses with a tool like Postman 

This generates the expected alert:

This is a simple example to showcase the captured HTTP request, source IP address and the User-Agent string. We can see the utility of a quick alert or signal that someone finds this url interesting (and by extension that someone had access to the place it was stored).

However, it is pretty clear by the hostname, that we are hitting a Canarytoken URL. Teams using our commercial Canary offering have the ability to use a custom domain. If you are using the free service, you can abstract away the Domain Name by using a DNS CNAME record.

We create a CNAME record in a domain we control. We want api.example.com to resolve to canarytokens.com. We add a CNAME record to point api.example.com to canarytokens.com, so that when the request is made, we still receive the alert!

Test:

curl http://api.example.com/ww0mayx9rro4h2l3keazj0zdk/api/v1/endpoint

Alert:

We can see that we receive the same alert, even though the hostname was changed. 

Setup Two: Webserver On Canary

Canaries have a built-in feature that allows you to configure a Webserver on it with a single click. With another click you can choose one of the default HTTP Skins. This is a great way to create an enticing server that attackers may attempt to interact with. Below is a basic example of applying the Jenkins Login HTTP Page Skin. Enable Webserver settings in the Canary Console:

When someone browses to the page they will see this.

(By default this exists to capture the credentials an attacker enters – and lets you know that somebody is messing around where they shouldnt be). A Skin here is simply a set of files, styles, scripts and images that you want to present to a visitor to the page. You can also upload custom skins to create any style you want. But! Even having a basic Canary-website like this allows us to capture REST API request attempts. You can post a reference to the service and usage. This could be in Github or other places with script snippets or command documentation. These are basic examples, but we think you get the idea. Below is a modified example of a REST API that allows you to transfer money between accounts. Place this example near the Accounting or Finance developers and lets see what happens.

Test:

curl http://srv02.example.com/api/v1/transfers \

  -u sk_test_4eC39HqLyjWDarjtT1zdp7dc: \

  -d amount=530 \

  -d currency=usd \

  -d destination=acct_acdcD82eZvKYlo2C8675309 \

  -d transfer_group=ORDER_66 \

  -v

Alert:

When anyone attempts to use this command, we will get the following alert!

In the above example, we only see the basic attributes of the request. (Canary won’t log the request

parameters by default). To capture the POST parameters, we add a custom file. Then alert would include

more context: how much money was attempted to be transferred and to which account!

For more detail on these parameters see this blog post about custom webroot configurations. In order to capture the POST parameters, we created a Defined Skin, you can read further about this here, Capturing Incident Data on POSTs.

Setup Three: Canarys Custom TCP Service 

In addition to the built-in HTTP Page Skins, Canaries allow you to create a Custom TCP Service that listen on a port you define, and gives you control over various interactions. This allows you to mimic services that may reflect websites or a REST API without using the standard HTTP service (80,443). If you would like to learn more about enabling and configuring a Custom TCP Service, be sure to review be sure to review our Service Configuration steps.

We can expand on the basic examples above.

We enable the Custom TCP Service on port 8443 in the Canary Console.

To make sure we don’t easily alert for false positives, we can configure the service to only alert when

TCP Connections send a string we are expecting by adding it to the Alert String field.

Test:

curl -X POST \

-H ‘X-Auth-Token:15dd7c486d81899f64643d6618c47a4efeedacdc’ \

http://srv02.example.com:8443/api/v1/resetpwd

Alert:

As we can see, our alert was triggered when the X-Auth-Token string in the Header matched our

configured Alert string. The Alert string is not required, if you are interested in alerting on any

connection attempts and logging the data sent, then leave the Alert string setting off.

An additional benefit of a Custom TCP Service is we can create tokens that are context specific.

The request URI, consists of: {URI-scheme} :// {URI-host} / {resource-path} ? {query-string} .

Given this is the case, we could create query strings in the request that identify the token.

curl -G -H  “Authorization: Bearer 15dd7c486d81899f64643d6618c47a4efeedac” \

http://srv02.example.com:8443/api/v1/resetpwd/subscriptions \

-d “api- version=2014-04-01-preview” \

-d “location=ThisisinStevesAzureADAuthenticationscript”

We never had to create a specific token, we can infer from the Request the key location dynamically. We don’t have to be as obvious in the parameter value either. We simply place these examples in links or in scripts. When the request arrives we receive an alert with the complete HTTP Request.

Placement:

One last topic needs to be considered here. How would an attacker find these examples, and what might

entice them to use them? That is a great question. There are a number of places where details for how to

connect to a REST API might show up. Much like opening a document with a Canarytoken or a URL

Canarytoken. We want to find ways that would cause someone to interact with our fake REST API. Here are a few thoughts and ideas. We would love to hear more about what you come up with as well.

Idea One: Place the REST API Canarytoken example in an Internal code repo, disguised as a  comment,

or a reply or a code snippet. 

Idea Two: Place a link in an internal page that calls out to the specific URL, or a file share, or laptop,

that looks like a developers workspace. There are a number of additional references and resources that

may also be useful to think about how your organization may find helpful.

We found the blog below from Detectify to be a helpful resource to get teams started thinking about how

an attacker might find and use a REST API endpoint or keys they discover.

Source:  https://labs.detectify.com/2021/08/10/how-to-hack-apis-in-2021/

Conclusion 

We can leverage already existing Canarytokens and Canary HTTP services to receive and alert

on unauthorized REST API request attempts. With the already existing HTTP primitives teams can extend or embed these examples within their organization and applications to detect and alert on unauthorized use.

References:

API Examples:

https://docs.github.com/en/rest/billing#about-the-billing-api

https://api.nasa.gov/

https://stripe.com/docs/api

https://docs.microsoft.com/en-us/rest/api/azure/

https://betterprogramming.pub/the-anatomy-of-an-http-request-728a469ecba9

https://labs.detectify.com/2021/08/10/how-to-hack-apis-in-2021/

Hacking APIs ,Breaking Web Application Programming Interfaces – by Corey Ball.

The AWS API Key Canarytoken (paid and free) is a great way to detect attackers who have compromised your infrastructure. The full details are in a previous blogpost, but in short: 

1. You go to https://canarytokens.org and generate a set of valid AWS API credentials;
2. Simply leave those in ~/.aws/config on a machine that’s important to you
3. Done!

If that machine is ever breached, the sort of attackers who keep you up at night will look for AWS API credentials, and they will try them. 

And when they do, we let you know that you’ve been breached.

When you receive an email/SMS/Slack message letting you know that the AWS API key that you left only on BuildServer-7 in Server-room #12 just got used to login to AWS, you know you have a problem. 

The underlying Canarytoken infrastructure relies on AWS APIs logging their own execution to CloudTrail. This lets us identify: which IP made the call; which API was executed (including both the service name and function); plus other details about the client executing the call.

The effectiveness of the AWS API Key Canarytoken shouldn’t be underestimated. Attackers have to try them; they could be the keys to the victim’s kingdom. If they don’t try the keys, they might be missing a golden opportunity. AWS API Keys are about the juiciest bait you can dangle in front of adversaries.

For defenders, the keys we supply are simple and entirely safe. They’re not tied to anything owned by the defender (the infrastructure sits completely at Thinkst), the keys have no permissions so they can’t be used to do anything, and no agents or software needs to be installed. You simply drop a text file, and wait to be notified if anything happens.

Between Canarytokens.org and our commercial Canarytokens offering, thousands and thousands of  machines worldwide have AWS API Canarytokens lying in wait for attackers.

Can AWS Canarytokens be detected?

This introduces a new goal for the super stealthy attacker. If they find an AWS API key on a server, can they tell if it’s a Canarytoken (or if it is a legitimate key?). Our view is that this only matters if an attacker can determine this without triggering the token. In other words, the Canarytoken fails if attackers can come across API credentials, and perform some test that returns whether or not the credentials are Canarytokens but defenders are never notified of the test. 

The key (heh) to making this happen as an attacker, would be to find AWS APIs that don’t log their execution, and also reveal information about the calling API Key to the caller.

For our purposes, there are four classes of error responses from AWS APIs:

1. Logs to CloudTrail, reveals no information about the API key
2. Logs to CloudTrail, does reveal account details from the API key
3. No CloudTrail logs, reveals no information about the API key
4. No CloudTrail logs, does reveal account details from the API key

As defenders,  4 is the worst case. If an API fulfills number 4, then Canarytokens can be detected. 3 is not great either, but softens the impact to “attackers can determine where credentials are valid, but not whether they’re Canarytokens”.

The next bit may surprise those who’ve never worked with it: the AWS API is a mishmash of response codes, error code, error strings, exception names, and data formatting. It is anything but consistent.

In the past, it’s been the case that a tiny subset of API errors did fulfill the constraints for class 4. The folks at Rhino Security blogged about this (RIP Spencer). While AWS (very slowly) does seem to react to reports on APIs which don’t show up in CloudTrail, that’s not sufficient. As it stands, we’re not aware of class 4 error responses currently in the AWS API, but at the rate at which new APIs are added, that’s little solace for thinking about the future.

There are certainly class 3 error responses in the current AWS API if you go looking hard enough.

In summary, the AWS API has a small number of endpoints which don’t log their own usage to CloudTrail, creating a blindspot for defenders. What do we do in those instances?

IAM Credential Reports

Scott Piper pointed out that Amazon actually does provide a backup mechanism for identifying credential usage. The IAM service lets you generate and download a report for all your credentials. This is a CSV file where each row belongs to an IAM user, and some of the columns identify when an access key was used, and on which AWS service it was used.

With the credential report, there are no longer class 3 or 4 error responses in a practical sense; we can also check the credential report to see if an API key was used, so no longer need to just rely on CloudTrail.

The report has three main drawbacks.

1. The report can only be generated once every four hours, so at worst there’s a four hour delay between credentials being used, and you seeing the notifications. 
2. The fidelity of the reports is low; you only can tell the last time the key was used and on which service (e.g. ec2, iam, sts, etc). You can’t tell which function was called. 
3. You don’t get any client information, including IP addresses or client versions.

However in spite of these drawbacks, it’s a huge step up in terms of reliability of this token type. Attackers don’t have places to sneakily test their API keys. This safety net means that every key usage will be detected and alerted on, albeit sometimes with lower fidelity artifacts. 

Even if you didn’t know the attackers IP Address, would it be worth knowing that the key that you placed on \\code-sign-22 was used to login to AWS this morning? Absolutely!

In fact, if a key is only used on what was previously a class 3 or 4 API, that’s even more of a signal since it implies the attacker is actively trying to avoid detection.

So we built an AWS safety net, to help us catch attackers who actively try to avoid detection.

Building an AWS Safety Net

The architecture is very straight forward. An AWS Lambda function fires on a periodic schedule. It pulls in the credential report from IAM, iterates over all the rows, and determines whether the last recorded use of the key happened more recently than the CloudTrail logs show. If so, the Safety Net kicks in, and sends out the alert:

Browsing to the history page for the incident, you see an annotation telling you that the Safety Net picked up this usage:

The Safety Net was deployed to our commercial customers a little while back thanks to Jason, and we’ve recently rolled it out to the free users on Canarytokens.org too.

Wrapping Up

AWS API logging previously left a small but significant gap that potentially gave attackers a way to use Canarytoken API keys without triggering alarms. With the deployment of the Safety Net, this gap has been plugged.

To see a World in a Grain of Sand Rice

– William Blake mh

If you are on TikTok (or a fan of talk shows) at the moment then, no doubt, your feed has included coloured rice being tossed in the air in the form of song lyrics, beloved cartoon characters, and even famous faces.

@mr.riceguy

Whilst coloured rice is not a new thing (for most preschool teachers, it is a cheap and effective way to keep kids entertained), a bunch of TikTok-ers have made a living off turning this simple play-thing into a full-on career. And, obviously, when a current trend is well-suited to our logo, we have to give it a go. Here’s how we got there:

What we used

• Rice
  • Our whole logo only needed 500g, however, we needed a few attempts to get it right and ended up using about 3kg. White rice is reusable, a jumble of multi-coloured rice, not so much…
• Food colouring
• Vinegar
  • 1 teaspoon per cup of rice
• Large (relatively stiff) rice tossing surface
  • We used a shelf from our cupboard…

How we did it

• We started by colouring the rice
  • We mixed 4 cups of rice with 4 teaspoons of vinegar and added food colouring until we got to the colour we wanted. The vinegar helps the colour spread evenly and get absorbed by the rice, so that, when playing with it, the colour doesn’t stain your fingers.

• Once mixed well (and to your colour-liking), we then dried the rice on trays lined with paper towel. The thinner the layer, the quicker it dries. Ours took about 2 hours.

• We scaled and sketched our logo onto the board
  • If we had had a projector, this would have been a much quicker + easier process, but alas.
  • Note that the image that you sketch out on the board will be reversed when tossed into the air. We drew the logo on in the correct orientation and then simply flipped the footage in editing

• Practice makes perfect
  • We had a few attempts with random shapes, just using white rice, so that we could practise the flip motion (and then could reuse the white rice)
  • We found the trick was to not raise the board too aggressively, but drop it as quickly as possible

“TEST”

Ultimately…

Verdict