NSA Surveillance: A Cost/Benefit Analysis

The U.S. government has spied on American citizens before. In 1918, 250,000 American volunteers joined the American Protective League (APL) to root out German spies residing in the United States during World War I. As David R. Henderson and I wrote in our book, Making Great Decisions in Business and Life:

These ordinary APL volunteers were very effective at getting information about American citizens and reporting suspicious behavior to the Justice Department. They cleverly found that they could “gain access to any house, on the grounds of checking their gas and electric service.” What was the result of their efforts? They reported well over a million “subversives,” which resulted in some deportations for immigration violations and some number of men caught in the midst of marital affairs, but not a single conviction for espionage. Most of the people held in prison for weeks without any charges filed had simply been critics of the war or had made the mistake of eating sauerkraut or listening to Beethoven.1

There probably were German spies in America, but those spies were the proverbial needles in the haystack. A key element in the efficacy of surveillance is the accuracy of the methods used.

Consider the Transportation Security Administration (TSA). In tests of TSA employees’ accuracy, agents found 40 percent of the bombs present on passengers and in luggage.2 It’s hard to find a needle in a haystack with this type of accuracy and, if one tries, the result will likely be a handful of hay and not a single needle.

Back to that other alphabet agency, the NSA. The question of NSA spying is intertwined with the issue of terrorism. If terrorism poses a huge threat to Americans, then spying might be justified. But, in reality, how dangerous is terrorism? Not very, according to John Mueller:

Even with the September 11 attacks included in the count, the number of Americans killed by international terrorism since the late 1960s (which is when the State Department began counting) is about the same as the number of Americans killed over the same period by lightning, accident-causing deer, or severe allergic reaction to peanuts.3

Still, many Americans persist in their fear of flying even though, statistically, one September 11-like disaster would need to occur each month for the risk of flying to equal the risk of driving—something we do daily without much concern.4 And a September 11-like hijacking is much less likely now that flight crews and passengers have learned that passivity equals death.

In addition to airplane hijackings, other possible terrorist techniques are also ineffective against a large number of Americans. For all the fear they induce in the population, radiological, chemical, and biological bombs are unlikely to kill large numbers of people.5

The benefit of the NSA’s surveillance is a reduction in the small probability of high-cost events. Ironically, the NSA’s spying activities introduce other low-probability, high-cost events, including the following:

1. 1. People with evil intentions, such as spammers, scam artists, hackers, and members of foreign governments, might eventually tap into and abuse the data the NSA has meticulously collected and organized.

2. 2. The NSA itself might abuse this sensitive data: we already know that at least a dozen NSA employees have abused secret surveillance programs in the past decade, most often to spy on their significant others.6

3. 3. The data could be used for political advantage. The IRS recently harassed various Tea Party groups. Indeed, the IRS has been repeatedly used for political persecution since at least FDR’s presidency.7 If the IRS can be used as a political weapon, so can the NSA.

4. 4. Freedom of the press could be at stake. In May, the Justice Department admitted that it secretly seized phone records from The Associated Press.8 Certainly, it’s not hard to imagine the NSA handing over personal information about reporters to the Justice Department or even intimidating reporters who write critically about the NSA or the government as a whole.

5. 5. The NSA’s spying might lead to anti-American reactions abroad—some of which has already happened after the spying on German Chancellor Angela Merkel—hurting this country both politically and economically.

6. 6. Revelations about NSA spying could spur the splintering of the Internet. American and foreign companies, along with numerous foreign governments, are alarmed at the depth of the U.S. government’s penetration into the Internet and the cell phone network. Concern about the possible abuse of this power has spurred talk of a “splinternet” in which the Internet is Balkanized.9 In the vanguard of this effort is Brazilian President Dilma Rouseff, who was surprised to learn that the NSA had been reading her private emails.10

Moreover, this leaves out one of the biggest costs of NSA spying, a cost that is not merely probable, but certain: A government agency is spying on us. The loss in privacy for Americans is a huge cost.

These low-probability collateral costs and the one certain cost are far from being worthwhile if the benefits are small.

And the benefits are small. Here’s why.

Assume that each completed terrorist act has, on average, a cost of $1 billion and that half the attempted acts are successfully completed. Also assume that, on average, finding a terrorist beforehand provides the government with a 20-percent chance of preventing an attack. Why just 20 percent? Because not all terrorists will act on their impulses, and the government has a poor record of preventing specific criminal acts. The marginal value of finding a terrorist is $100 million ($1 billion times 50 percent times 20 percent).

The cost of surveillance includes perhaps half of the NSA’s budget. (Later, I’ll try to bias the finding in favor of NSA surveillance by assuming a much lower number.) The size of that budget, though classified, is estimated to have been $10.8 billion in 2013.11 The government will need extra investigative time and effort to follow up with all the people flagged, correctly or incorrectly, as terrorists (subpoenas, wiretaps, extra surveillance, perusing emails, stakeouts, arrests, jail time, etc.), and those who are arrested will need legal representation to exonerate themselves. Assume that the total cost per suspected terrorist is $30,000. For the NSA’s surveillance programs to make sense, the marginal value per terrorist should be greater than the marginal cost per terrorist.

The NSA’s surveillance budget per terrorist, plus follow-up costs for those suspected, should equal less than the $100 million per terrorist value calculated above. Let’s employ some useful insights and equations from the medical profession.

Many thousands of people walking the streets today harbor deadly diseases such as cancer. Luckily, some diagnostic tests might detect these cancers early enough, enabling physicians to treat and, perhaps, cure some of these people. Why are they left untested? Shouldn’t everyone be screened for every type of cancer? It turns out that the problem is complicated.

Every diagnostic test has two key measures of accuracy: the accuracy of detecting a disease that is present in a person (sensitivity) and the accuracy of not detecting a disease that is absent in a person (specificity).

We know that, among the general population, finding those individuals harboring cancer is like finding a needle in a haystack, and our ability to do that depends critically on the ratio of needles to hay. Is there one needle for every 100 pieces of hay or one for every 100 million pieces of hay? Every “false positive” incurs the cost and risk of further evaluating healthy people.

Positive predictive value12 (PPV), which is derived from Bayes’ theorem, is a good measure of the ratio of the true positive and false positive rates, and it equals the proportion of persons who test positive who really have the disease. A PPV of 10 percent means that out of every 10 individuals who test positive, one really has the condition and nine do not. Medical experts say that for a screening test to be considered a good idea for the general population, it should have a positive predictive value of 5 to 25 percent.13 For reference, the PPV for mammography for women 50 and older is 14 percent, while the prostate cancer PSA test has a better PPV of 20 to 50 percent.

Assume that there are 1,000 terrorists on American soil. For a test that has a sensitivity and specificity of 40 percent (assuming TSA-like values), the PPV is a miserable 0.00021 percent.

PPV = sensitivity × prevalence / ((sensitivity × prevalence) + (1 − specificity) × (1 − prevalence))

Prevalence = 1 thousand / 317 million = 0.000003155

PPV = 0.4 × 0.000003155 / ((0.4 × 0.000003155) + (1 − 0.4) × (1 − 0.000003155))

PPV = 0.000002103

For every true terrorist uncovered, the NSA would incorrectly flag 475,50014 innocent American residents. With the NSA budget of $5 million per terrorist ($5 billion divided by 1,000) and a follow-up cost of $30,000 per accused, the marginal cost per real terrorist is over $14 billion, which is far above the marginal value per terrorist of $100 million. Notice that even if one assumes a zero budget for the NSA, the marginal cost per real terrorist is still over $14 billion. The big driver of costs is not the NSA budget, but the cost of incorrectly flagging 475,500 innocent American residents for every true terrorist.

To put this into perspective, if the goal were to find all 1,000 terrorists, 475.5 million innocent people would be flagged as terrorists, and the total follow-up costs would equal $14 trillion. There are not enough American residents to accuse—there are only 317 million of us—and the U.S. GDP is only $15.7 trillion. Simply accusing everyone in the country would achieve the same result and allow the federal government to save $5 billion per year on the NSA’s budget. In other words, if the NSA persists in using inaccurate tests that, because of their high false-positive rates, end up accusing every American of being a terrorist, then we can simply accuse everyone right away and avoid the suspense and hassle.

Perhaps the NSA’s tests are more accurate. With a sensitivity and specificity of 80 percent and the other assumptions unchanged, the PPV is 0.00126 percent. That means that for every true terrorist uncovered, the tests would incorrectly flag 79,250 innocent Americans. The total follow-up costs would be almost $2.4 billion, which is still far above the value per terrorist of $100 million. The NSA budget per terrorist would still be rounding error.

The numbers don’t look good, even with the unrealistically high values of 98 percent for sensitivity and specificity. The PPV would be 0.0155 percent, and 6,470 innocents would be snagged for each terrorist. The follow-up and budget costs per terrorist would total $200 million, which is double the value per terrorist of $100 million.

For the NSA’s spying program to make sense, the collateral costs arising from the spying would need to be close to zero; the value per terrorist would need to be higher (say $250 million); there would need to be more terrorists on U.S. soil (say 5,000); the follow-up costs would need to be lower per suspect (say $14,000); and the NSA’s tests would need to be highly accurate (say 80 percent). These assumptions are unrealistic.

My guess is that there are 100 or fewer terrorists on American soil. Why? If we consider the lousy success rate for the TSA’s screening, combined with the lack of airplane hijackings and bombings over the past decade, we come to the conclusion that there just aren’t that many motivated terrorists in our society.

Could the NSA really be twice as accurate as the TSA? The TSA has the luxury of physically processing all fliers through specially designed screening systems at airports. The NSA has, arguably, a more difficult job, snatching electronic snippets out of the ether.

The NSA’s screening of the general population to find terrorists is analogous to the medical profession’s screening for cancer. Using the medical profession’s insights and equations, we can see that the NSA is foisting a poor screening tool on Americans. Even putting concomitant costs and the huge loss of privacy aside, the NSA’s electronic spying costs substantially more than it is worth and, on cost/benefit grounds, should be terminated.