As the Snowden leaks continue to dribble out, it has become increasingly obvious that most nations planning for “cyber-war” have been merely sharpening knives for what looks like an almighty gunfight. We have to ask ourselves a few tough questions, the biggest of which just might be:
“If the NSA was owning everything in sight (and by all accounts they have) then how is it that nobody ever spotted them?”
The Snowden docs show us that high value targets have been getting compromised forever, and while the game does heavily favour offence, how is it possible that defence hasn’t racked up a single catch? The immediate conclusions for defensive vendors is that they are either ineffective or, worse, wilfully ignorant. However, for buyers of defensive software and gear, questions still remain.
The last dump, published by The Spiegel on the 17th of January went by pretty quietly (compared to previous releases) but the PDFs released contain a whole bunch of answers that might have slipped by un-noticed. We figured it would probably be worth discussing some of these, because if nothing else, they shine a light on areas defenders might have been previously ignoring. This is especially true if you play at a level where nation state adversaries form part of your threat model. (and, as the leaks show, NSA targets commercial entities like mobile providers, so it’s not just the domain of the spooks.)
The purpose of this post isn’t to discuss the legality of the NSA’s actions or the morality of the leaks, what we are trying to answer is:
“Why did we never see it coming?”
We think that the following reasons help to explain how this mass exploitation remained under the radar for so long:
- Amazing adherence to classification/secrecy oaths;
- You thought they were someone else;
- You were looking at the wrong level;
- Some beautiful misdirection;
- They were playing chess & you were playing checkers;
- Your “experts” failed you miserably.
1. Amazing adherence to classification/secrecy oaths;
The air of secrecy surrounding the NSA has been amazingly impressive and until recently, they had truly earned their nickname of “No Such Agency.” A large number of current speakers/trainers/practitioners in infosec have well acknowledged roots in the NSA. It was clear from their skill-sets and specialities that they were obviously involved with CNE/CNO in their previous lives. If one were to probe deeper, one could make even more guesses as to the scope of their previous activities (and by inference we would have obtained a low resolution snapshot of NSA activities)
 |
 |
 |
|
Dave Aitel
Fuzzing & Exploit
frameworks
|
Jamie Butler
Rootkits & Memory
Corruption
|
Charlie Miller
Fuzzing & Exploitation
|
Reading through the Snowden documents, a bunch of “new” words has been introduced into our lexicon. Interdiction was relatively unheard of, and the the word “implant” was almost never used in security circles, but has now fairly reliably replaced the ageing “rootkit”. We have read the documents for a few hours and have adopted these words, but ex-NSA’ers have clearly lived with these words for years of their service. That the choice of wording has not bled far beyond the borders at Fort Meade is interesting and notable. It is an amazing adherence to classification and secrecy, deserves admiration and has likely helped the NSA keep some of its secrets to date.
(This is to be expected when innovation occurs out of sight, terminology diverges. When GCHQ cryptographers conceived early public-key crypto they called it “non-secret cryptography”, however this was only revealed many years after “public-key” had become commonplace. Now that “implant” is in the public domain (and is associated with NSA), there seems little reason for vendors to continue with “rootkit”.)
2. You thought they were someone else;
Skilled adversaries operating under cover of a rioting mob is hardly a new tactic, and when one considers how much “bot” related activity is seen on the Internet, hiding amongst it is an obviously useful technique. The dump highlights two simple examples where the NSA leverages this technique. Performing “4th party collection” we essentially have the NSA either passively, or actively stealing intelligence from other intelligence agencies performing CNE.
The fact that the foreign CNE can be parasitically leeched, actively ransacked or silently repurposed, means that even attacks that use malware belonging to country-X, using TTP’s that strongly point to country-X could just be activity that should be attributed to the 4th party collection program.
Of course theres no need for the NSA to limit themselves to just making use of foreign intelligence agencies. Through DEFIANTWARRIOR you see them making active use of general purpose botnets too. With some details on how botnet hijacking works (sometimes in coordination with the FBI) their slides also offer telling advice on how to make use of this channel:
This raises two interesting points that are worth pondering. The first (obvious) one, is that even regular cybercrime botnet activity could be masking a more comprehensive form of penetration and the second is how much muddier it makes the waters of attribution.
For the past few years, a great deal has been made of how Chinese IP’s have been hacking the Western World. When one considers that the same slide deck made it clear that China had by far the greatest percentage of botnets, then we are forced to be more cautious when attributing attacks to China just because they originated from Chinese IP’s. (We discussed our views on weakly evidenced China attribution previously [here] & [here]).
3. You were looking at the wrong level;
A common criticism of the top tier security conferences is that they focus on attacks that are overly complex, while networks are still being compromised by un-patched servers and shared passwords. What the ANT catalogue and some of the leaks revealed, is that sensitive networks have more than enough reason to fear complex attacks too. One of the most interesting documents in this regard appears to be taken from an internal Wiki, cataloguing ongoing projects (with calls for intern development assistance).
The document starts off strong, and continues to deliver: “TAO/ATO Persistence POLITERAIN (CNA) team is looking for interns who want to break things. We are tasked to remotely degrade or destroy opponent computers, routers, servers and network enabled devices by attacking the hardware using low level programming.”
For most security teams, low level programming generally means shellcode and OS level attacks. A smaller subset of researchers will then aim at attacks targeting the Kernel. What we see here, is a concerted effort to aim “lower”:
“We are also always open for ideas but our focus is on firmware, BIOS, BUS or driver level attacks.”
The rest of the document then goes on to mention projects like:
- “we have discovered a way that may be able to remotely brick network cards… develop a deployable tool”.
- “erase the BIOS on a brand of servers that act as a backbone to many rival governments”
- “create ARM-based SSD implants.”
- “covert storage product that is enabled from a hard drive firmware modification”
- “create a firmware implant that has the ability to pass to and from an implant running in the OS”
- “implants for the newest SEAGATE drives..”, “for ARM-based Hitachi drives”, “for ARM-based Fujitsu drives”, “ARM-Samsung drives”..
- “capability to install a hard drive implant on a USB drive”
- “pre-boot persistence.. of OSX”
- “EFI module..”
- “BERSERKR is a persistent backdoor that is implanted into the BIOS and runs from SMM”
All of this perfectly aligns with the CNO/GENIE document which makes it clear that base resources in that project:
“will allow endpoint implants to persist in target computers/servers through technology upgrades, and enable the development of new methodologies to persist and maintain presence within hard target networks”.
We have worked with a few companies who make religious use of whitelisting technologies and have dealt with some who would quickly discover altered system files on sensitive servers.
We know a tinier subset of those who would verify the integrity of running hosts using offline examination but organizations that are able to deal with implanted firmware or subverted BIOSes are few and far between. In the absence of hardware based TPM’s, this is currently a research grade problem that most people don’t even know they have.
4. Some beautiful misdirection;
Even if we were completely underprepared as defenders, one would think that those cases where implants were communicating back to the NSA would have been discovered (even if by accident) sooner or later. Once more, the documents reveal why this would not have resulted in the classic “smoking gun”.
A common IR process when an attack has been discovered is to determine where the exfiltrated data is going to. In the most simplistic case (or if big budget movies are involved) this simple step could allow an analyst to say:
“The data from this compromised host is going to HOST_B in country_X. So country_X is the culprit.”
Of course, since even spirited teenagers have been making use of “jump hosts” since the 90’s, a variation on this would be not just to base the attribution on the location of HOST_B, but to observe who then accesses HOST_B to “collect the loot”. (It’s the sort of time you really want to be the “global passive adversary”).
Even this would have tipped the NSA’s hand sooner or later, and what we see from the docs is a clever variation on the theme:
We see the use of an entire new protocol, called FASHIONCLEFT to effectively copy traffic off a network, attach metadata to it, then hide the packet within another packet allowed to exfil the targeted network.
Tunnelling one type of traffic over another is not novel (although a 27 page interface control document for the protocol is cool) but this still leaves open the possibility that you would see victim_machine talking to HOST_X in Europe. This is where passive collection comes in..
This is beautiful! So the data is munged into any packet that is likely to make it out of the network, and is then directed past a passive collector. This means that we cant rely on the host the data was sent to for attribution, and even if we did completely own the last hop, to see who shows up to grab the data, we would be watching in vain, because the deed was done when the packets traversed a network 3 hops ago.
This really is an elegant solution and a beautiful sleight of hand. With the NSA controlling tens of thousands of passive hosts scattered around the Internet, good luck ever finding that smoking gun!
(in their own words)
5. They were playing chess & you were playing checkers;
Whats very clear from the breadth of the information is just how out of their depth, so many of the other players at this table are. Many are hilariously outgunned, playing on a field that has already been prepared, using tools that have already been rigged… and whats worse, is that many of them don’t even know this.
In 2010, we gave a presentation at the CCDCOE in Estonia to NATO folks. Our talk was titled: “Cyberwar – Why your threat model is probably wrong!” (The talk has held up relatively well against the revelations, and is worth a quick read, even though it predated the discovery of STUXNET)
One of the key take aways from the talk (aside from the fact that any expert who referred to DDoS attacks when talking about cyberwar should be taken with a pinch of salt) was that real attackers build toolchains. Using examples from our pen-testing past, we pointed out how most of the tools we built went into modular toolchains. We mentioned that more than anything else, robust toolchains were the mark of a “determined, sponsored adversary”.
Our conclusions from the talk were relatively simple:
The nature of the game still heavily favours offence, and attacker toolchains were likely much more complex than the “sophisticated attacks” we had seen to date. When you look at the Snowden documents, if there is one word they scream, its toolchains:
If there are two words, its “sophisticated toolchains“
The USA (and their Five Eyes partners) were clearly way ahead of the curve in spotting the usefulness of the Internet for tradecraft and, true to the motto of U.S cyber command have been expending resources to ensure “global network dominance“. While organizations all over the world have struggled over the past few years to stand up SoC’s (security operations centers) to act as central points for the detection and triage of attacks, the documents introduce us (for the first time) to its mirror image, in the form of a ROC:
From [media-35654.pdf]:
In terms of ROC capacity, the documents show us that in 2005, the ROC was 215 people strong running a hundred active campaigns per day – In 2005! (thats generations ago in Internet years).
In an op-ed piece we penned for Al Jazeera in 2011, we mentioned that nation states following the headlines about the US training tons of cyber warriors (with the CEH certification of all things) would be gravely mistaken, that offensive capability had been brought to bear on nation-states, long before the official launch of US Cybercom and these docs validate those words.
In fact, if you are a nation state dipping your toes in these waters, its worth considering the documented budgets for project GENIE which we mentioned earlier. With an admittedly ambitious stated goal to “plan, equip and conduct Endpoint operations that actively compromise otherwise intractable targets” we can guess that project GENIE would cost a bit.
Fortunately, we don’t have to guess, and can see that in 2011, GENIE alone cost $615MM, with a combined headcount of about 1500 people.
JUST. FOR. PROJECT. GENIE.
Of course while debate rages about the morality of governments buying 0days (and while some may think this is a new concept) the same document shows that back in 2012, about $25MM was set aside for “community investment” & “covert purchases of software vulnerabilities”. $25MM buys a whole lot of 3rd party 0day.
The possibly asymmetric nature of cyberwar means that small players are able to possibly punch above their weight-class. What we see here, is proof positive that the biggest kid in the room has been working on their punching for a long time…
6. Your “experts” failed you miserably.
The snowden leaks crossed over from infosec circles into the global zeitgeist which meant international headlines and soundbytes on CNN. This in turn has led to a sharp rise in the number of “CyberWar Experts” happy to trot out their opinions in exchange for their 15 minutes of fame..
VC funding is rushing to the sector and every day we see more silver bullets and more experts show up… but, it would behoove us to pause for a bit to examine the track records of these “experts”.
How did they hold up against yesterdays headlines? I have seen 6 figure consultants trying to convince governments that 0days are never used and have seen people talk of nation state hacking with nothing more than skinned metasploit consoles and modern versions of back-orrifice.
How many of the global “threat intelligence” companies are highlighting TTPS actually in use by APEX predators (instead of merely spotting low hanging fruit). If they are not, then we need to conclude that they are either uninformed or complicit in deluding us and either option should cap the exorbitant fees many currently seek.
Conclusion?
The leaks give us an insight into the workings of a well refined offensive machine. The latest files show us why attributing attacks to the NSA will be difficult for a long time to come and why “safe from nation state adversaries” requires a great deal more work, by people who are qualified to do so..
If nothing else, the leaks reiterate the title from our 2010 talk.. “Cyberwar.. why your threat model is probably wrong”
[If you enjoy this update, you really should subscribe to ThinkstScapes]
14 comments On If the NSA has been hacking everything, how has nobody seen them coming?
I enjoyed reading this and want to point out the NSA mission is INTELLIGENCE collection, specifically SIGINT. To point out what was stated about terminology the US military men and women who are assigned there go to work every day to protect the US, and 'access is the essence of espionage'. An agency created well before the the idea of an Internet the NSA has the extremely difficult job of gaining access to intelligence and communications used by military organizations skilled enough to keep even more difficult with the Internet. They're not an offensive 'attacker', which for clarity would be the Cyber Command and if anyone would like to dig a lot of the NSA felt posed an issue for the General over both since one has the mission of gaining and keeping access and the the the other a warrior organization. Snowdan was compromised by the Russians and is a spy, not a whistle blower, not a Manning but an individual who was aware the military secrets passed along would damage the US and our close allies and while the men and women working on damage control who probably are probably aware of information about the weaknesses and resources to address keep their silence the people they are working to defend are being pitted against them by classic Soviet methods Putin practices and learned while in the KGB. The only thing I disagree with is about the experts being wrong but this is based on their being experts in infosec and not intelligence. The military networks and communications using the network technologies commercial organizations use (i.e., the internet) aren't 'secure' from a military perspective but their threatened by capable military groups who have penetrated the CIA to it's core. NSA don't arrest people, they don't spend their time looking at people in their own country (i don't care about any links where someone says someone looked into some girlfriend) – it's not their mission or job and my guess is they didn't sign up to do that or they would have applied for a job with TSA or some other cop group. Good points made and the first time I've commented on this subject the links to the NSA controlling thoughts and a few shots were it.Eddie Mastering Laziness (almost there)
With encrypted communication becoming commonplace it gets harder to distinguish legitimate traffic from illegitimate.
We did see it coming, and we watch it happen. Your opening sentence is wrong: “how has nobody seen them coming”. It was not “nobody” by a long margin. Keep in mind that the NSA also employs a team tasked with deliberately discrediting the people and their stories when sightings got aired. Just because some “Anonymous” blog comment pokes ridicule, doesn't mean the original story is any less true.Case in point: why did they use “=” instead of “^=” when they added the FIPS PRNG to GPG? Infiltration is staring everyone in the face, and people do shout about it… Nobody notices or cares though. They've been trained to think everyone who points this stuff out is a hysterical conspiracist, and ignore.
I'll say it again, the attacks on the non-US based file hosting company Mega.com are simply an extension of this now proven aim of keeping the data flow and storage where the NSA and other US players can easily delve and tinker and assess. Google, via YouTube, infringes more than Kim Dotcom ever has. But because they play on the NSA team (& indeed are funded partly by them) crazy copyright laws aren't used and abused against them.
The problem is that a lot of what's in the documents you quote is pure fiction, and with that much noise, it's easy for the signal to get lost.Back on topic: BIOS and firmware-level attacks were somewhat common in the late 80s and early 90s. At that point, when the Internet took rise, probably around the advent of Windows 95, it became easy enough for “hobbiest” hackers to forget about the more difficult task of infiltrating hardware and focus on the software. But clearly the NSA had the resources and inclination to keep focus on the lowest level.It isn't actually that difficult to create logic testers to verify proper operation of hardware… they're used for troubleshooting, and by factories for QC before a product even goes out the door. What seems important, now, is that these tests be made available to the public with open APIs. That would, at least, allow researchers to identify infections in the wild, and from there potentially develop software-based identification techniques.But it's important to remember that all is not lost. Even on rogue routers, rogue traffic can be identified by sniffing actual outgoing traffic — which is as easy as physically splitting the signal, a la a hub — and comparing it with intended/known outgoing traffic. Any difference means something is awry. This isn't (to my knowledge) something that's being done yet, but I suspect traffic validation and verification will become more common, and it will be interesting to see what's discovered.
Physical access to network and computing equipment might be gained during the shipping process. The units you purchased are intercepted and replaced with almost identical units with something extra. Your vendor's warehouse might be compromised. No need for a spy agency to have physical access to your site when they can target the hardware before it arrives.
You did a good job on your guesses. Enough that I wrote a small essay [1] on it on Schneier's blog where I post my designs and reviews. This didn't all happen in a vacuum: others and I issued warnings for decades based on academic research, NSA's own standards, and field work. High assurance field advocated rigorous methods at every layer with simple, strong TCB as being minimal to stop nation states. We were ignored (mostly) and results speak for themselves. My essay includes the risks framework I used in my security work. Would've prevented all non-physical and non-INFOSEC attacks in TAO catalog. [1] https://www.schneier.com/blog/archives/2015/02/understanding_n.html#c6688978Nick PSecurity Engineer/Researcher(High assurance focus)
Why did “we” never see this coming? For the same reasons “no-one” saw the 2008 financial crash coming, and the same reason people are largly denying the financial crash which we are on the brink of.Lots of people have been warning of what was coming for many years. They really got going as soon as Admiral Poindexter released the logo for his “Total Information Awareness” programme in 2002. Many of the programmes begun under that umbrella are exactly the programmes Snowden blew the whistle on. And yet Poindexter was absolutely open about his intentions. The warnings mainly came in two forms. First in articles and blogs on the internet from people labelled as “conspiracy theorists”. These are people labelled, put in a box and ignored by most of “we”. The other source of the warnings came in the form of propaganda pieces (arguably), such as the BBCs The Last Enemy.So lots of people saw it coming. “We” either chose not to listen, or were too busy being “entertained”.
Well… as far as I can tell the word “implant” was used from NSA Red Team from 2012 (before the leaks)See this video https://www.youtube.com/watch?v=HnnvVnsDCGw (at 6:47), and also this video https://www.youtube.com/watch?v=aoG1XzUk7sU (at 13:54). The NSA Red Team uses the word “implant” there. Two times in these two videos.Also in the 2nd video see at 10:23, 11:26, 15:26, and 17:13. You may want to add a fourth person next to the 3 guys you already have (ps I really love Raffi). Also on the 1st video he appears lot of times.
We probably should not be surprised. Doing things that involve multiple forms of deception, corruption of otherwise legitimate entities, coercion, etc. are all hallmarks of intelligence operations going back centuries. The NSA is merely applying the same principles to cyberspace.The bit about exfiltrating data that is subsequently picked up by passive collectors is another example of how the NSA leverages its privileged position on the Internet – due to much of the Internet passing through US routers run by US corporations. This enables the NSA to do things that a lot of other, even major, nation states would have difficulty doing due to their non-privileged status on the Internet. Not to mention relative budgets and available technical expertise (although as noted, that isn't a high bar.) What is a high bar is precisely the NSA's privileged status on the Internet.And the NSA's goal has clearly been the same as the US government (and US corporate) goal from almost day one: total hegemony over the planet. While everyone has been worrying about the “rise of China” or “Russian aggression” or even a fanciful Islamic “caliphate”, the real threat to world peace and security remains…the United States. The fundamental belief system of the United States elites is that total control means total security – for them – no matter what it costs in lives, economies, or civil rights for the world and for the US citizen.And until the US electorate wakes up and realizes that “we have seen the enemy – and they is us” – nothing will change.
http://www.whale.to/b/nsa7.htmlhttp://www.whale.to/b/akwei.pdfhttp://www.bibliotecapleyades.net/scalar_tech/esp_scalartech12.htmhttp://www.legco.gov.hk/yr05-06/english/bc/bc52/papers/bc52cb2-2607-1e-scan.pdfheres a perfect example of one of the first people to call them out, far before anyone knew who this little snowden pansy was
For number 3 I guess it's good that there are companies like PrivateCore and LegbaCore now actually looking at firmware level attacks, eh? But I mean it's not like people haven't been talking about them since 2006 or so. Hell, BIOS level attacks were mentioned in the 2005 Hoglund & Butler rootkits book…
Hardware Implants … if you have those = physical access and if you have physical access everything else is relatively easy.
For your number two, yes, that's the classic goal of counterintelligence: to takeover the enemy C2. For number four, ncovert was released in 2003 perhaps predating FASHIONCLEFT. Don't forget Unisys Stealth, a network technology that makes Tor and I2P look like toys. Perhaps the difference between implants and rootkits is that implants include the stack to call home with the knowledge that a passive collector somewhere will be able to pick up on the flipped bits?