We’ve previously discussed how Canarytokens can detect when your website has been cloned and used in phishing campaigns. We also released an Azure Entra ID Login token that can be used to detect this activity on your tenant’s Entra ID Login page.
Today, we’re taking that concept a step further by introducing an automated response pipeline that detects phishing attempts, correlates which of your users fell victim to the attack and takes immediate action to minimise the impact of the phishing attack.
The Challenge of Modern Phishing
Adversary-in-the-Middle (AitM) attacks have raised the stakes in phishing. The technique allows attackers to bypass many traditional security measures, including some forms of multi-factor authentication. In an AitM attack, the adversary positions themselves as a real-time proxy between the user and the legitimate website, in this case, your Entra ID Login page.
When a user attempts to log in, the attacker’s server intercepts the login request, forwards it to your tenant’s Entra ID Login page, and then relays the response to the user.
This setup allows the attacker to capture usernames and passwords, session cookies, and even one-time passcodes used in multi-factor authentication. Unaware of the intermediary, the user believes they interact directly with the legitimate site, making these attacks particularly sneaky and effective.
While detecting these attacks is crucial, it’s only the first step. The real challenge lies in what comes next: You’ve detected a phish, now what?
Bridging the Gap: From Alert to Action
Canaries and Canarytokens exist to give you a strong signal of badness. This post shows how we can build a pipeline translating those strong signals into immediate protective actions.
An Automated Phishing Response Pipeline
We will illustrate the automated response pipeline with example snippets of PowerShell code. This type of response can be deployed in various ways, from your own Azure Runbook to response logic built into your SIEM/SOAR. The main objective of this post is to provide you with an overview of the process and relevant code that can be reused and made applicable in your particular environment.
Your response to the Entra ID Login token alert will likely start with a webhook trigger from your Canary Console. The response will run code correlating user accounts compromised by the phishing attack and initiate defensive actions to prevent attackers from getting more deeply entrenched.
Firstly, we’d like to limit the alerts that trigger the response pipeline to Entra ID Login token alerts. Since the response pipeline deals specifically with phishing attacks, it does not need to receive other alerts. To achieve this, we can do the following:
- Create a new flock that will only be used for our Entra ID Login tokens.
- Add our response pipeline webhook to the flock
The pipeline consists of five key steps:
- Canarytoken Alert: The Entra ID Login token alert is triggered and pushed to the response pipeline using a Webook.
- Alert Ingestion and Phishing Domain Identification: The phishing server’s domain is extracted from the alert data received by the webhook.
- IP Address Correlation: The system resolves the phishing domain to its associated IP addresses, creating a list that can be used to identify sign-in requests originating from the phishing server.
- User Session Discovery: Leveraging the Microsoft Graph API, we identify user sessions that may have been compromised by querying for sign-in requests originating from the identified IP addresses.
- Automated Mitigation: Restrict further sign-in attempts from the identified IP addresses, revoke the existing sign-in sessions and invalidate all refresh tokens associated with those users, effectively blocking and locking out the attacker.
By automating these five steps, this pipeline provides a rapid and effective response to detected phishing attempts, significantly reducing the window of opportunity for attackers and enhancing the organisation’s overall security posture.
We’ve included a diagram to illustrate the main process flow with this automated response pipeline.
You will note that some delays are built into the flow, and we make additional queries to the Canary Console API. We’ve noted that there can be several minutes of delay between a user signing in and Azure audit logs being populated for that event. Since the Entra ID Login token alert will trigger before Azure can update the Sign-In logs, we need to build a retry and delay mechanism.
The additional queries to the Canary Console API are required to identify any other phishing domains that attackers may have used after the initial Webhook had been triggered. The alert data contained in the Webhook only pertains to the first Entra ID Login token being triggered; subsequent events (within a 60-second window) will be rolled up and associated with that initial event. However, since the Webhook has already been triggered, we need to query the API to retrieve the rolled-up events.
Let’s dive into each step and cover a few relevant code snippets in PowerShell.
Canarytoken Alert
The process begins when the Canarytoken alert is triggered. This alert is sent to the response pipeline via a webhook and contains information about the potential phishing attempt, including the URL of the cloned site.
This code snippet shows how the webhook data is ingested and parsed to extract key information, such as the cloned site URL.
param (
[object]$WebhookData
)
$data = $WebhookData | ConvertFrom-Json
$clonedSite = $data.AdditionalDetails | Where-Object { $_[0] -eq "Cloned Site" } | ForEach-Object { $_[1] }Alert Ingestion and Phishing Domain Identification
Once the alert data is received, we extract the domain of the phishing server from the cloned site URL. This allows us to identify the source of authentication requests the phishing server relays.
if ($clonedSite) {
# Extract hostname from "Cloned Site" URL
$uri = [System.Uri]$clonedSite
$hostname = $uri.Host
Write-Output "URI: $uri"
Write-Output "Hostname: $hostname"
} else {
Write-Output "No cloned site hostname found in the webhook data."
}Once we have the phishing server’s hostname, we can use it in subsequent steps to identify potentially compromised user sessions.
IP Address Correlation
With the phishing server hostname identified, we resolve it to obtain the associated IP addresses. This step creates a list of IP addresses that can be used to identify sign-in requests originating from the phishing server.
$ipAddresses = [System.Net.Dns]::GetHostAddresses($hostname) | ForEach-Object { $_.IPAddressToString }User Session Discovery
Many of the subsequent code snippets will show requests to the Azure Graph API. This will need an Access Token for authentication. The following request can be used to obtain a token.
function Get-GraphApiAccessToken {
param (
[Parameter(Mandatory=$true)]
[string]$TenantId,
[Parameter(Mandatory=$true)]
[string]$ClientId,
[Parameter(Mandatory=$true)]
[string]$ClientSecret
)
$resource = "https://graph.microsoft.com/"
$authUrl = "https://login.microsoftonline.com/$TenantId/oauth2/v2.0/token"
$body = @{
grant_type = "client_credentials"
client_id = $ClientId
client_secret = $ClientSecret
scope = "$resource/.default"
}
try {
$tokenResponse = Invoke-RestMethod -Method Post -Uri $authUrl -ContentType "application/x-www-form-urlencoded" -Body $body
return $tokenResponse.access_token
}
catch {
Write-Error "Failed to obtain access token: $_"
return $null
}
}This code queries the Azure Graph API for sign-in logs, filtering by the IP addresses associated with the phishing attack. It then compiles a list of unique users who have successfully authenticated from the phishing server IP addresses.
foreach ($ip in $clonedSiteIPAddresses) {
Write-Host "Querying auditLogs for signIns from IP: $ip"
# Define the query URL
$queryUrl = "https://graph.microsoft.com/v1.0/auditLogs/signIns?`$select=userPrincipalName,userId,createdDateTime,ipAddress&`$filter=ipAddress eq '$ip'"
# Query the Microsoft Graph API
$headers = @{
"Authorization" = "Bearer $accessToken"
"Content-Type" = "application/json"
}
$results = Invoke-RestMethod -Method Get -Uri $queryUrl -Headers $headers
# Get unique users from the results
$uniqueUsers = $results.value | Select-Object -Property userId, userPrincipalName -Unique
# Add unique users from this IP to the overall list $allUniqueUsers
foreach ($user in $uniqueUsers) {
if ($user.userId -and -not $allUniqueUsers.ContainsKey($user.userId)) {
$allUniqueUsers[$user.userId] = $user.userPrincipalName
}
}
}
Automated Mitigation
We can initiate mitigating responses to minimise the impact of the phishing attack. This will include three specific actions:
- Restrict further sign-in attempts from the identified IP addresses
- Revoke the existing sign-in sessions
- Invalidate all refresh tokens associated with those users
We can restrict further sign-in attempts by adding the IP address(es) associated with the phishing server to a Named Location, which we will then associate with a Conditional Access Policy. This policy will prohibit access to resources from future sign-in attempts originating from the phishing server.
This approach allows you to:
- Maintain a single named location for all phishing server IPs.
- Add new IPs to this location as needed.
- Have a single Conditional Access policy that blocks access from all IPs in this Named Location.
Here be dragons. A particularly crafty attacker may create DNS entries that point to your legitimate corporate IPs and attempt to trigger the Entra ID Login token with these. If your response pipeline automatically adds the IP addresses without further inspection, this could inadvertently block access to your cloud resources. As such, it’s important to create an allow-list with known good IP addresses (such as your VPN or corporate egress address range) that can be used to filter out any legitimate IP addresses before adding them to a Named Location and/or invalidating user sessions originating from that source.
We’ll start by creating a Named Location (PhishingServers). You will need to specify at least 1 IP address range. Since we’ll be adding IP addresses to this programmatically, we’ll add 127.0.0.1/32 temporarily to get by the UX restriction on Azure. The temporary IP address can be deleted when our first phishing server IP addresses are added.
Next, we’ll create a Conditional Access Policy. You will need to configure the following on the Conditional Access Policy.
- Users – Select the users or group of users that this Policy will apply to
- Target Resources – Define which target resources the Policy will apply to
- Network – Select the PhishingServers Named Location that was created earlier
- Conditions – This will reference the PhishingServers Named Location
- Grant – Block access
If you have not previously applied any custom Conditional Access Policies, you will need to “disable security defaults”. Security defaults provide a basic level of protection for all users. When disabled, you’re responsible for implementing and managing your own security measures, which means you’ll need to configure and manage security settings previously handled automatically manually. This allows for more granular control over security policies but requires additional overhead to execute correctly. Implementing Conditional Access policies also requires Azure AD Premium P1 or P2 licenses.
You will need the ID associated with the created Named Location to continue. While retrieving the ID through the Azure Portal UI does not appear possible, it can be obtained with the following API request.
function Get-NamedLocations {
param (
[Parameter(Mandatory=$true)]
[string]$AccessToken
)
$headers = @{
"Authorization" = "Bearer $AccessToken"
"Content-Type" = "application/json"
}
$listUri = "https://graph.microsoft.com/v1.0/identity/conditionalAccess/namedLocations"
$namedLocations = @()
do {
$response = Invoke-RestMethod -Method GET -Uri $listUri -Headers $headers
$namedLocations += $response.value
$listUri = $response.'@odata.nextLink'
} while ($listUri)
foreach ($location in $namedLocations) {
[PSCustomObject]@{
ID = $location.id
DisplayName = $location.displayName
Type = $location.'@odata.type'.Split('.')[-1]
CreatedDateTime = $location.createdDateTime
ModifiedDateTime = $location.modifiedDateTime
}
}
}With the Named Location and Conditional Access Policy created, we go back to the automated response. We’ve already obtained the IP addresses associated with the phishing server and a list of users that had authenticated from those IP addresses, which means we can now:
- Add the phishing server IP addresses to the Named Location to restrict future sign-in attempts.
- Revoke active sign-in sessions for users who authenticated through the phishing server.
- Invalidate refresh tokens associated with those users.
This effectively blocks and locks out the attacker, mitigating the potential impact of the phishing attack.
A PATCH request is made to the conditional access endpoint on the Microsoft Graph API to add the IP address ranges associated with the phishing server.
function Add-IPToNamedLocation {
param (
[Parameter(Mandatory=$true)]
[string]$NamedLocationId,
[Parameter(Mandatory=$true)]
[string]$IPAddress,
[Parameter(Mandatory=$true)]
[string]$AccessToken
)
$headers = @{
"Authorization" = "Bearer $AccessToken"
"Content-Type" = "application/json"
}
# Get existing named location details
$getUri = "https://graph.microsoft.com/v1.0/identity/conditionalAccess/namedLocations/$NamedLocationId"
$existingLocation = Invoke-RestMethod -Method GET -Uri $getUri -Headers $headers
# Prepare the new IP range
$newIpRange = @{
"@odata.type" = "#microsoft.graph.iPv4CidrRange"
"cidrAddress" = if ($IPAddress -match '/') { $IPAddress } else { "$IPAddress/32" }
}
# Add the new IP range to the existing ones
$updatedIpRanges = $existingLocation.ipRanges + $newIpRange
$body = @{
"@odata.type" = "#microsoft.graph.ipNamedLocation"
"ipRanges" = $updatedIpRanges
} | ConvertTo-Json -Depth 4
$updateUri = "https://graph.microsoft.com/v1.0/identity/conditionalAccess/namedLocations/$NamedLocationId"
$updateResponse = Invoke-RestMethod -Method PATCH -Uri $updateUri -Headers $headers -Body $body
Write-Host "Successfully added IP address $IPAddress to named location $NamedLocationId"
}
We then revoke all sign-in sessions associated with the identified users and invalidate associated refresh tokens.
function Terminate-UserSessions {
param (
[Parameter(Mandatory=$true)]
[string]$UserId,
[Parameter(Mandatory=$true)]
[string]$AccessToken
)
$headers = @{
"Authorization" = "Bearer $AccessToken"
"Content-Type" = "application/json"
}
# 1. Revoke sign in sessions
$revokeSessionsUri = "https://graph.microsoft.com/v1.0/users/$UserId/revokeSignInSessions"
$revokeSessionsResponse = Invoke-RestMethod -Method POST -Uri $revokeSessionsUri -Headers $headers
# 2. Invalidate refresh tokens
$invalidateTokensUri = "https://graph.microsoft.com/v1.0/users/$UserId/invalidateAllRefreshTokens"
$emptyBody = "{}" # Empty JSON object
$invalidateTokensResponse = Invoke-RestMethod -Method POST -Uri $invalidateTokensUri -Headers $headers -Body $emptyBody
Write-Host "Session termination process completed for user $UserId"
}
# Process all unique users across all IPs
foreach ($userId in $allUniqueUsers.Keys) {
$upn = $allUniqueUsers[$userId]
Write-Host "Revoking sessions for user: $upn (ID: $userId)"
Terminate-UserSessions -UserId $userId -AccessToken $accessToken
}
To better understand why we need to both revoke sign-in sessions and invalidate refresh tokens, we’ll need to look more closely at what those actions do.
Refresh tokens are used to obtain new access tokens without requiring the user to re-authenticate. Revoking these prevents users from obtaining new access tokens using their existing refresh tokens. However, this doesn’t immediately invalidate existing access tokens, which may remain valid for their full lifetime (typically 1 hour). Users with active sessions may continue to access resources until their current access tokens expire.
Revoking sign-in sessions is intended to terminate all active sessions for the user across all devices and applications. This is more comprehensive than just revoking refresh tokens, as it aims to invalidate all types of tokens associated with the user’s sessions. However, its effectiveness can be limited by factors such as caching mechanisms in certain applications, propagation delays across Microsoft’s services, and the possibility that some specialised or legacy applications might not immediately recognise the revocation.
Once the phishing server IP address has been added to the Named Location, subsequent sign-in attempts from other users will receive this message upon completing authentication.
Implementation Considerations
There are a couple of important considerations that should be kept in mind when implementing an automated response pipeline, such as the one detailed in this post.
- The Canary Console alert will trigger before Azure has synced SignIn logs, and this delay needs to be accounted for.
- A trigger from a webhook will only contain the initial incident data. If other incidents are generated for the same token within a set time window (the event horizon), those events will be aggregated into the original alert. To retrieve the aggregated events, the Console API should be queried to retrieve additional incidents associated with the original alert. The event horizon for Azure Entra ID Login tokens is currently set to a 60-second rolling window and can be customised by reaching out to Thinkst Support.
- The Azure Graph API is queried to identify users who signed in from the IP address associated with the phishing server; those users’ sessions are then invalidated. Subsequent queries to the Graph API to identify additional users will also return the previously identified users. As such, the already processed users should be tracked so that only newly identified user sessions are invalidated.
- Beware of crafty attackers creating DNS entries pointing to your legitimate corporate IPs to trigger Entra ID Login tokens. Automatic addition of these IPs could inadvertently block access to cloud resources. Implement an allow-list of known good IPs to filter legitimate addresses before adding to Named Locations or invalidating user sessions from that source.
Additional Defensive Measures
Automating these steps provides a rapid and effective response to detected phishing attempts, significantly reducing attackers’ window of opportunity. Beyond sign-in restrictions and session invalidation, many additional actions can be taken:
- User Password Resets: Initiate password resets for the affected users and include them in the next round of phishing awareness training.
- Logging and Reporting: Additional logging and reporting can be performed to detail which actions were taken against user accounts, the phishing domain and associated IP addresses.
- Phishing Domain Takedown: The Canarytoken alert and Azure Entra Sign-In Logs provide clear evidence of phishing. A takedown report can then be sent to the domain registrar or hosting provider requesting its removal to prevent further exploitation.
Conclusion
The Cloned Site Canarytokens are powerful tools for detecting phishing attacks. This post outlines just one possible way to extend its capabilities, moving from detection to automated response.
By leveraging the alerts generated by Canarytokens, organisations can build custom response pipelines that suit their specific needs and infrastructure. The automated response to phishing that we’ve described here is an example of how Canarytokens can be integrated into a broader security strategy, enhancing an organisation’s ability to detect and respond swiftly to potential threats.
The strength of Canarytokens lies in their flexibility and ease of deployment. Whether used for breach detection or as part of a more comprehensive automated response system, they provide a valuable early warning system in a package that attackers can’t resist.