Almost famous: behind the scenes of a feature that didn’t make the cut

Introduction

A counterintuitive truth is that great products are defined by both the features they include, as well as those they don’t. We spend a lot of time pondering potential new features for Thinkst Canary to make sure the added value exceeds the inevitable cognitive complexity that new features (or new UX elements) bring. This post will dive into a recent Labs research effort that we ended up leaving on the cutting room floor.

Background

We are always on the lookout for attacker techniques that can be repurposed for defense. A likely candidate emerged when skimming the recent CISA red team reports (1, 2): Abusing Unconstrained Kerberos Delegation in Active Directory networks.

Active Directory (AD) allows for ease of use through single-sign-on (SSO). A user who’s logged into their workstation can navigate internal file-shares, web applications, and databases without having a separate user account to manage (or another password to remember). The user’s machine works with the AD domain controller (DC) and either presents an authentication hash, or, preferably, uses the Kerberos protocol to mutually authenticate to a server. SSO with Kerberos simplifies the user interaction, and prevents a host of hijacking or adversary-in-the-middle attacks, since both the client and server are authenticated. 

Kerberos delegation comes into play when one server wants to access resources on behalf of the end user. An example from Microsoft’s documentation is an intranet web application that renders data from an SQL database. Access control is simplified when the web application can only see files that the end-user has permissions to. Legacy configurations of AD initially supported this with a concept called “Unconstrained Delegation”. A server with this permission can act on behalf of any user authenticated to it, to any other resource in the network. 

This is as dangerous as it sounds.

If the application has been subverted, it can then “be” any user who’s connected to it, including Domain Administrators. For this reason, Microsoft has introduced constrained variants of delegation in which the delegation is limited to only certain resources or services. These settings, including Unconstrained Delegation are a property in a Computer’s object that is editable inside the AD Users and Computers tooling:

Naive first stab

Attackers know Unconstrained Delegation is a desirable target, and there are many tools¹ that look for (and exploit) it. Our [naive] first idea was to manually set this property on a Canary’s machine account in AD in order to attract the attention of tools that scan for Unconstrained Delegation. This would point attackers towards the Canary–profit!. Since Canary is a honeypot that doesn’t store real data or credentials, we reasoned that it would be manageable from a risk perspective. 

Unfortunately, that’s not true in this case. 

An attacker would now have additional targets: the Canary’s AD object and the machine password the Canary uses to authenticate to the DC. If the Canary’s AD object has Unconstrained Delegation set, anyone with the Canary’s machine password can act as the Canary within the AD and leverage the Unconstrained Delegation privileges. In other words, the Canary’s machine password would give attackers a powerful step-up towards Domain Admin. 

The machine password exists on the Canary itself (this is integral to the concept of being domain-joined) and this means that Canary would now hold a valuable secret. This is contrary to one of our central design goals: not storing valuable data on the Canary–there should be nothing an attacker will gain by compromising a bird. 

Under the naive approach (and without special permissions), if the attacker had the Canary password (or ownership over its AD object), they could create AD object attributes that direct clients towards their system². The attacker could then induce a DC to connect to them, allowing them to impersonate the DC’s machine account–granting full administrator privileges. This attack sequence is detailed beautifully by Dirk-jan, but in short, (if we followed this approach) by controlling a Canary, its machine AD object, or the user that created it (or can take over ownership of it), an attacker gains a high-confidence path to Domain Admin. 

At this point, we had built a firmer understanding of the attack space (and how dangerous Kerberos Delegation is–even Microsoft Defender alerts on it now), but couldn’t countenance putting our Canary users in such a risky spot. We were about to move on when we had another thought…

Attempt number two

In our initial reading, we came across a great blog by Crowe, where they lay out the risks of Unconstrained Delegation. What if we could create a machine object in AD with Unconstrained Delegation that doesn’t actually have a backing machine–in short a kind of Ghost Server? Then we could use the attack setup steps from the Crowe blog to point attackers looking for the Ghost Server to seamlessly connect to a Canary. This approach has a few benefits over our initial (naive) approach:

• We can ensure that the Ghost Server object is owned by a Domain Admin, and the ACLs on the object prevent any other groups or users from modifying its object attributes. Since the Canary doesn’t have any changes to its AD object, it remains as banal an attack target as before.
• There is no domain-joined machine that can be compromised to recover the machine account password from. It only exists on the DC. Since this machine account is never used, there won’t be copies of its credentials in-memory on any system other than the DC.
• The Ghost Server will pop up when any attackers are looking for machines with Unconstrained Delegation, but the attackers will actually be connecting to a Canary.

In order to execute this deception, we created the Ghost Server object, and pointed its DNS at a Canary. As expected, the Ghost Server pops up as a juicy target for attackers, but any attackers are then pointed to the Canary. This was close to perfect, however, Kerberos authentication would fail as the DC would issue a ticket for the Ghost Server, and since Kerberos is a mutual authentication protocol, the Canary couldn’t pretend to be the Ghost without having access to its valuable secret.

To bypass this limitation, we had to dig deeper into Service Principal Names, or SPNs. SPNs are like DC-internal records for the Kerberos service to map a service name (e.g., ghost-server.domain.local) to a machine account (e.g., ghost-server). Even though we had changed the DNS records, the Kerberos service on the DC would issue a ticket for the non-existent machine. Somewhat counter-intuitively, if we removed all of the SPNs (using AD’s setSPN.exe) from the Ghost Server’s machine account, and added the Ghost Server’s domain names to the Canary’s account in the SPN records, it worked! 

As shown in the figure, when an attacker looks for machines with Unconstrained Delegation, the AD object for the Ghost Server is returned. When the attacker’s machine tries to connect to the Ghost Server, the DNS directs it to the Canary. Then, when the attacker’s machine asks the DC for a Kerberos session with the Ghost Server, the DC helps initiate a secure connection with the Canary’s machine account. 

So, we now have a nifty approach that puts up a bright flag for attackers, but just as they charge towards it, we pull it away and in its place there’s… a Canary. This perfectly embodies our Canary mindset of Valuable, not Vulnerable. The Ghost Server appears to be a valuable attack target, but since the Ghost Server doesn’t actually exist, it’s not vulnerable. 

But, it still doesn’t make the cut… 

Why we didn’t get the W

In short: this feature would be a win for many of our customers, but adds risk to some, and differentiating between those two requires nuance. For many networks, the careful addition of the Ghost Server object will add no more risk, but there certainly will be networks where it would. Environments with complex and transitive group inheritance, or AD networks that change over time (such as during an M&A integration) could result in more users having control over the Ghost Server’s object–thus opening the aperture for malicious privilege escalation.

We firmly believe that everyone should deploy Canaries, and they will add value without adding defender cognitive load. The Canary deployment process and console interactions are designed to reduce as much friction as possible to get birds out into the wild where they can alert on badness. The frictionless process we strive for is at odds with the slower, deep analysis needed to determine if the Ghost Server approach could safely work in a specific environment³.

Do it Yourself (if you’re careful!)

Just because it doesn’t make a great Canary feature doesn’t mean it’s worthless. The technique can be effectively used if the blue-team is circumspect with its use. For that reason we’re releasing this post and an open-source script to:

1. Create a Ghost Server AD object (though we suggest not naming it Ghost Server 😛)
2. Configure the Ghost Server’s AD object to support Unconstrained Delegation and restrict its ACLs to writable only by Domain Admins
3. Point the Ghost Server’s DNS at an IP or machine (CNAME) of your choosing (i.e., the domain-joined honeypot–we recommend Canary 👍)
4. Add SPN records for the Ghost Server and associated network services to the machine account of the honeypot

To see this in action, we first run this script with Domain Admin permissions:

Then, when an attacker enumerates machines with unconstrained delegation, they find GhostServer, if they were to scan that domain (or its IP), they find the Canary’s services:

If they browse to that server in Explorer, they see files from the Canary, even though the hostname in the location bar is the Ghost Server:

Finally, if the attacker tries to open one of these files, the defender gets an alert!

It’s important to treat the Ghost Server machine account as any other Domain Admin or DC account. Monitor it for changes, and ensure when any organizational changes are made to AD that the account ownership doesn’t get inherited by other OUs or groups. This technique can help Canaries stand out to attackers, but with the caveats that if that account is compromised, there is a path to Domain Admin. (It is also worth noting that a very savvy attacker could become suspicious of the DNS/SPN records and avoid the honeypot because of it). Consider your own environment and take some time to weigh the risks and rewards. If nothing else, the spelunking was worth it.

Conclusion

Deciding what to leave out of a product is sometimes as valuable as deciding what to add in. This blog post aimed to reveal a bit of the behind-the-scenes on how we looked at a potential deception capability, almost wrote it off too early, then came up with a neat, almost-good-enough solution. 
As we’ve highlighted in the past, making something too easy to click through (or copy-and-paste) can overly-influence users who are rushing to complete their tasks. Considering this, we feel that this feature has too much potential downside risk to enable by default. It’s cute, and useful, but remains an activity best left for blue-teamers with ample time and resources to weigh deploying.

1. Such as: krbrelayx, pywerview, and Impacket ↩︎
2. DNS records point clients to the server, Service Principal Names (SPNs) help the DC set up the authentication between AD users and services. ↩︎
3. Tools like BloodHound are great for defenders to get an attacker view of their network, and help make sure the Ghost Server object isn’t mutable by any surprising users. ↩︎