As the Snowden leaks continue to dribble out, it has become increasingly obvious that most nations planning for "cyber-war" have been merely sharpening knives for what looks like an almighty gunfight. We have to ask ourselves a few tough questions, the biggest of which just might be:
"If the NSA was owning everything in sight (and by all accounts they have) then how is it that nobody ever spotted them?”
The Snowden docs show us that high value targets have been getting compromised forever, and while the game does heavily favour offence, how is it possible that defence hasn't racked up a single catch? The immediate conclusions for defensive vendors is that they are either ineffective or, worse, wilfully ignorant. However, for buyers of defensive software and gear, questions still remain.
The last dump, published by The Spiegel on the 17th of January went by pretty quietly (compared to previous releases) but the PDFs released contain a whole bunch of answers that might have slipped by un-noticed. We figured it would probably be worth discussing some of these, because if nothing else, they shine a light on areas defenders might have been previously ignoring. (This is especially true if you play at a level where nation state adversaries form part of your threat model. (and, as the leaks show, NSA targets commercial entities like mobile providers, so it’s not just the domain of the spooks.)
The purpose of this post isn’t to discuss the legality of the NSA's actions or the morality of the leaks, what we are trying to answer is:
"Why did we never see it coming?"
We think that the following reasons help to explain how this mass exploitation remained under the radar for so long:
- 1. Amazing adherence to classification/secrecy oaths;
- 2. You thought they were someone else;
- 3. You were looking at the wrong level;
- 4. Some beautiful misdirection;
- 5. They were playing chess & you were playing checkers;
- 6. Your "experts" failed you miserably.
1. Amazing adherence to classification/secrecy oaths;
The air of secrecy surrounding the NSA has been amazingly impressive and until recently, they had truly earned their nickname of "No Such Agency." A large number of current speakers/trainers/practitioners in infosec have well acknowledged roots in the NSA. It was clear from their skill-sets and specialities that they were obviously involved with CNE/CNO in their previous lives. If one were to probe deeper, one could make even more guesses as to the scope of their previous activities (and by inference we would have obtained a low resolution snapshot of NSA activities)
Fuzzing & Exploit
Rootkits & Memory
Fuzzing & Exploitation
Reading through the Snowden documents, a bunch of "new" words has been introduced into our lexicon. Interdiction was relatively unheard of, and the the word "implant" was almost never used in security circles, but has now fairly reliably replaced the ageing "rootkit". We have read the documents for a few hours and have adopted these words, but ex-NSA’ers have clearly lived with these words for years of their service. That the choice of wording has not bled far beyond the borders at Fort Meade is interesting and notable. It is an amazing adherence to classification and secrecy, deserves admiration and has likely helped the NSA keep some of its secrets to date.
(This is to be expected when innovation occurs out of sight, terminology diverges. When GCHQ cryptographers conceived early public-key crypto they called it “non-secret cryptography”, however this was only revealed many years after “public-key” had become commonplace. Now that “implant” is in the public domain (and is associated with NSA), there seems little reason for vendors to continue with “rootkit”.)
2. You thought they were someone else;
Skilled adversaries operating under cover of a rioting mob is hardly a new tactic, and when one considers how much "bot" related activity is seen on the Internet, hiding amongst it is an obviously useful technique. The dump highlights two simple examples where the NSA leverages this technique. Performing "4th party collection" we essentially have the NSA either passively, or actively stealing intelligence from other intelligence agencies performing CNE.
The fact that the foreign CNE can be parasitically leeched, actively ransacked or silently repurposed, means that even attacks that use malware belonging to country-X, using TTP's that strongly point to country-X could just be activity that should be attributed to the 4th party collection program.
Of course theres no need for the NSA to limit themselves to just making use of foreign intelligence agencies. Through DEFIANTWARRIOR you see them making active use of general purpose botnets too. With some details on how botnet hijacking works (sometimes in coordination with the FBI) their slides also offer telling advice on how to make use of this channel:
This raises two interesting points that are worth pondering. The first (obvious) one, is that even regular cybercrime botnet activity could be masking a more comprehensive form of penetration and the second is how much muddier it makes the waters of attribution.
For the past few years, a great deal has been made of how Chinese IP's have been hacking the Western World. When one considers that the same slide deck made it clear that China had by far the greatest percentage of botnets, then we are forced to be more cautious when attributing attacks to China just because they originated from Chinese IP’s. (We discussed our views on weakly evidenced China attribution previously [here] & [here]).
3. You were looking at the wrong level;
A common criticism of the top tier security conferences is that they focus on attacks that are overly complex, while networks are still being compromised by un-patched servers and shared passwords. What the ANT catalogue and some of the leaks revealed, is that sensitive networks have more than enough reason to fear complex attacks too. One of the most interesting documents in this regard appears to be taken from an internal Wiki, cataloguing ongoing projects (with calls for intern development assistance).
The document starts off strong, and continues to deliver: "TAO/ATO Persistence POLITERAIN (CNA) team is looking for interns who want to break things. We are tasked to remotely degrade or destroy opponent computers, routers, servers and network enabled devices by attacking the hardware using low level programming.”
For most security teams, low level programming generally means shellcode and OS level attacks. A smaller subset of researchers will then aim at attacks targeting the Kernel. What we see here, is a concerted effort to aim "lower":
"We are also always open for ideas but our focus is on firmware, BIOS, BUS or driver level attacks."
The rest of the document then goes on to mention projects like:
- "we have discovered a way that may be able to remotely brick network cards... develop a deployable tool".
- "erase the BIOS on a brand of servers that act as a backbone to many rival governments"
- "create ARM-based SSD implants."
- "covert storage product that is enabled from a hard drive firmware modification"
- "create a firmware implant that has the ability to pass to and from an implant running in the OS"
- "implants for the newest SEAGATE drives..", "for ARM-based Hitachi drives", "for ARM-based Fujitsu drives", "ARM-Samsung drives"..
- "capability to install a hard drive implant on a USB drive"
- "pre-boot persistence.. of OSX"
- "EFI module.."
- "BERSERKR is a persistent backdoor that is implanted into the BIOS and runs from SMM"
All of this perfectly aligns with the CNO/GENIE document which makes it clear that base resources in that project:
"will allow endpoint implants to persist in target computers/servers through technology upgrades, and enable the development of new methodologies to persist and maintain presence within hard target networks".
We have worked with a few companies who make religious use of whitelisting technologies and have dealt with some who would quickly discover altered system files on sensitive servers.
We know a tinier subset of those who would verify the integrity of running hosts using offline examination but organizations that are able to deal with implanted firmware or subverted BIOSes are few and far between. In the absence of hardware based TPM's, this is currently a research grade problem that most people don’t even know they have.
4. Some beautiful misdirection;
Even if we were completely underprepared as defenders, one would think that those cases where implants were communicating back to the NSA would have been discovered (even if by accident) sooner or later. Once more, the documents reveal why this would not have resulted in the classic "smoking gun”.
A common IR process when an attack has been discovered is to determine where the exfiltrated data is going to. In the most simplistic case (or if big budget movies are involved) this simple step could allow an analyst to say:
“The data from this compromised host is going to HOST_B in country_X. So country_X is the culprit.”
Of course, since even spirited teenagers have been making use of "jump hosts" since the 90's, a variation on this would be not just to base the attribution on the location of HOST_B, but to observe who then accesses HOST_B to "collect the loot". (It's the sort of time you really want to be the "global passive adversary”).
Even this would have tipped the NSA's hand sooner or later, and what we see from the docs is a clever variation on the theme:
We see the use of an entire new protocol, called FASHIONCLEFT to effectively copy traffic off a network, attach metadata to it, then hide the packet within another packet allowed to exfil the targeted network.
Tunnelling one type of traffic over another is not novel (although a 27 page interface control document for the protocol is cool) but this still leaves open the possibility that you would see victim_machine talking to HOST_X in Europe. This is where passive collection comes in..
This is beautiful! So the data is munged into any packet that is likely to make it out of the network, and is then directed past a passive collector. This means that we cant rely on the host the data was sent to for attribution, and even if we did completely own the last hop, to see who shows up to grab the data, we would be watching in vain, because the deed was done when the packets traversed a network 3 hops ago.
This really is an elegant solution and a beautiful sleight of hand. With the NSA controlling tens of thousands of passive hosts scattered around the Internet, good luck ever finding that smoking gun!
(in their own words)
5. They were playing chess & you were playing checkers;
Whats very clear from the breadth of the information is just how out of their depth, so many of the other players at this table are. Many are hilariously outgunned, playing on a field that has already been prepared, using tools that have already been rigged... and whats worse, is that many of them don’t even know this.
In 2010, we gave a presentation at the CCDCOE in Estonia to NATO folks. Our talk was titled: "Cyberwar - Why your threat model is probably wrong!" (The talk has held up relatively well against the revelations, and is worth a quick read, even thought it predated the discovery of STUXNET)
One of the key take aways from the talk (aside from the fact that any expert who referred to DDoS attacks when talking about cyberwar should be taken with a pinch of salt) was that real attackers build toolchains. Using examples from our pen-testing past, we pointed out how most of the tools we built went into modular toolchains. We mentioned that more than anything else, robust toolchains were the mark of a "determined, sponsored adversary”.
Our conclusions from the talk were relatively simple:
The nature of the game still heavily favours offence, and attacker toolchains were likely much more complex than the "sophisticated attacks" we had seen to date. When you look at the Snowden documents, if there is one word they scream, its toolchains:
If there are two words, its "sophisticated toolchains"
The USA (and their Five Eyes partners) were clearly way ahead of the curve in spotting the usefulness of the Internet for tradecraft and, true to the motto of U.S cyber command have been expending resources to ensure "global network dominance". While organizations all over the world have struggled over the past few years to stand up SoC's (security operations centers) to act as central points for the detection and triage of attacks, the documents introduce us (for the first time) to its mirror image, in the form of a ROC:
In terms of ROC capacity, the documents show us that in 2005, the ROC was 215 people strong running a hundred active campaigns per day - In 2005! (thats generations ago in Internet years).
In an op-ed piece we penned for Al Jazeera in 2011, we mentioned that nation states following the headlines about the US training tons of cyber warriors (with the CEH certification of all things) would be gravely mistaken, that offensive capability had been brought to bear on nation-states, long before the official launch of US Cybercom and these docs validate those words.
In fact, if you are a nation state dipping your toes in these waters, its worth considering the documented budgets for project GENIE which we mentioned earlier. With an admittedly ambitious stated goal to "plan, equip and conduct Endpoint operations that actively compromise otherwise intractable targets" we can guess that project GENIE would cost a bit.
Fortunately, we don’t have to guess, and can see that in 2011, GENIE alone cost $615MM, with a combined headcount of about 1500 people.
JUST. FOR. PROJECT. GENIE.
Of course while debate rages about the morality of governments buying 0days (and while some may think this is a new concept) the same document shows that back in 2012, about $25MM was set aside for "community investment" & "covert purchases of software vulnerabilities". $25MM buys a whole lot of 3rd party 0day.
The possibly asymmetric nature of cyberwar means that small players are able to possibly punch above their weight-class. What we see here, is proof positive that the biggest kid in the room has been working on their punching for a long time…
6. Your "experts" failed you miserably.
The snowden leaks crossed over from infosec circles into the global zeitgeist which meant international headlines and soundbytes on CNN. This in turn has led to a sharp rise in the number of "CyberWar Experts" happy to trot out their opinions in exchange for their 15 minutes of fame..
VC funding is rushing to the sector and every day we see more silver bullets and more experts show up... but, it would behoove us to pause for a bit to examine the track records of these "experts".
How did they hold up against yesterdays headlines? I have seen 6 figure consultants trying to convince governments that 0days are never used and have seen people talk of nation state hacking with nothing more than skinned metasploit consoles and modern versions of back-orrifice.
How many of the global "threat intelligence" companies are highlighting TTPS actually in use by APEX predators (instead of merely spotting low hanging fruit). If they are not, then we need to conclude that they are either uninformed or complicit in deluding us and either option should cap the exorbitant fees many currently seek.
The leaks give us an insight into the workings of a well refined offensive machine. The latest files show us why attributing attacks to the NSA will be difficult for a long time to come and why “safe from nation state adversaries” requires a great deal more work, by people who are qualified to do so..
If nothing else, the leaks reiterate the title from our 2010 talk.. “Cyberwar.. why your threat model is probably wrong”
[If you enjoy this update, you really should subscribe to ThinkstScapes]