Signals intelligence and the management of military competition

In the past few years many observers have been alarmed by the high-tech realities of cyber-security, cyber-spying, and cyber-warfare. The current interest is on the apparent impunity with which government-sponsored intruders have managed to penetrate and exploit the computer systems of government and corporate organizations -- often extracting vast quantities of sensitive or classified information over extended periods of time. The Sony intrusion and the Office of Personnel Management intrusion represent clear examples of each (link, link). Gildart Jackson's Cyberspies: The Secret History of Surveillance, Hacking, and Digital Espionage provides a very interesting description of the contemporary realities of cyber-spying by governments and private intruders.

It is very interesting to realize that the cat-and-mouse game of using cryptography, electronic signals collection, and intelligence analysis to read an adversary's intentions and communications has a long history, and resulted in problems strikingly similar to those we currently face. A very good recent book that conveys a detailed narrative of the development of signals intelligence and cryptography since World War II is Stephen Budiansky's Code Warriors: NSA's Codebreakers and the Secret Intelligence War Against the Soviet Union. The book offers a surprisingly detailed account of the formation and management of the National Security Agency during the Truman presidency and the sophisticated efforts expended toward penetrating military and diplomatic codes since the Enigma successes of Bletchley Park.

There are several particularly interesting lessons to be learned from Code Warriors. One is a recognition of the remarkable resourcefulness and technical sophistication that was incorporated into the signals intelligence establishment in the 1940s and 1950s. Many of us think primarily of the achievements of Bletchley Park and the breaking of code systems like Enigma during World War II. But signals intelligence went far beyond cryptography. For example, a great deal of valuable intelligence resulted from "traffic analysis" -- specific information about time and location of various encrypted messages. Even without being able to read the messages themselves it was possible for analysts to draw inferences about military activity. This is an early version of meta-data analysis of email and phone calls.

Another surprise was the ability of intelligence establishment communications experts in the 1950s to use "side-channel" attacks to gain access to adversaries' communications channels (multi-channel radio teletype machines, for example). By recording the electromagnetic emissions, power fluctuations, and acoustic patterns of code machines, typewriters, and teletype machines it was possible to reconstruct the plain text that was passing through these devices.

Most interesting for readers of Understanding Society, however, are the large number of problems of organization, management, and leadership that effective intelligence service required. Several problems were particularly intractable. Inter-service rivalries were an enormous obstacle to effective collection, analysis, and use of signals intelligence. Motivating and retaining civilian experts as workers within a large research organization in the military was a second. And the problem of defending against misappropriation of documents and secrets by trusted insiders was another.

The problem of inter-agency rivalries and competition was debilitating and intractable. Army and Navy intelligence bureaus were enormously reluctant to subordinate their efforts to a single prioritized central agency. And this failure to cooperate and share information and processes led to substantial intelligence shortfalls.

The 1946 agreement between the Army and Navy to “coordinate” their separate signals intelligence operations had merely sidestepped glaring deficiencies in the entire arrangement, which was quickly proving itself unequal to the new technical and intelligence challenges they faced in attacking the Russian problem. (lc 1933)

But AFSA’s seventy-six-hundred-person staff and $35 million budget remained a small share of the total enterprise, and both the Army and Air Force cryptologic agencies continued to grab important projects for themselves. ASAPAC and USAFSS both duplicated AFSA’s work on Soviet and Chinese codes throughout the Korean War, and simply ignored attempts by AFSA to take charge of field processing within the theater. The Air Force had meanwhile established its headquarters of USAFSS at Brooks Air Force Base in Texas, a not too subtle attempt to escape from the Washington orbit altogether. (lc 2933)

AFSA was powerless to prevent even the most obvious duplication of effort: for over a year the Army and the Air Force both insisted on intercepting Russian and Chinese air communications, and it was not until March 1952, after months of negotiations, that ASA finally agreed to leave the job to the Air Force. The Navy meanwhile flatly refused to put its worldwide network of direction-finding stations—which provided the single most important source of information on the location and movement of Soviet surface ships and submarines—under central control. (lc 2949)

Also challenging was the problem of incorporating smart, innovative civilian experts into what had become rigid, hierarchical military organizations. Keeping these civilians -- often PhDs in mathematics -- motivated and productive within the strictures of a post-war military bureaucracy was exceptionally difficult. During WWII the atmosphere was conducive to innovative work:

At GC&CS and Arlington Hall in particular, formal lines of authority had never counted for much during the war; getting the job done was what mattered, and in large part because no one planned to make a career of the work, no one was very career-minded about office politics or promotion or pay or protecting their bureaucratic turf. Cecil Phillips remembered wartime Arlington Hall as a true “meritocracy” where a sergeant, who in a considerable number of cases might have a degree from MIT or Harvard or some other top school, and a lieutenant might work side by side as equals on the same problem and no one thought much about it. (lc 1417)

But after the war the bureaucratic military routines became a crushing burden:

At ASA, peace brought a flood of pettifogging orders, policy directives, and procedural instructions, accompanied by a succession of martinet junior officers who rotated in and out and often knew nothing about cryptanalysis but were sticklers for organization, military protocol, and the chain of command. Lengthy interoffice memoranda circulated dissecting the merits of developing a personnel handbook, or analyzing whether a proposed change in policy that would allow civilian employees of Arlington Hall to be admitted to the post movie theater was consistent with Paragraph 10, AR 210-389 of the Army Regulations. “Low pay and too many military bosses” would be a recurring complaint from ASA’s civilian workforce over the next few years, along with a sense that no matter how much experience they had or how qualified they were, the top positions in each division would always go to a less qualified Army officer. (lc 1430)

The problem of coordinating, directing, and managing these high-talent scientists proved to be an ever-challenging task for NSA as well:

Among the administrative nightmares of the explosively growing, disjointed, and highly technical top-secret organization that Canine inherited was a notable lack of skilled managers. That was a failing common to creative and technical enterprises, which always tended to attract people more at home dealing with abstract ideas than with their fellow human beings, but it was especially acute in the very abstract world of cryptanalysis. “I had a terrible time finding people that could manage,” Canine related. “We were long on technical brains at NSA and we were very short on management brains.” 50 The splintering of the work into hundreds of separate problems, each isolated technically and for security reasons from one another, exacerbated the difficulties of trying to assert managerial control on an organization made up of thousands of individualistic thinkers who marched to no identifiable drum known to management science. (lc 3582)

And of course the problem of insider spying turned out to be essentially insurmountable, from the defection of NSA employees William Martin and Bernon Mitchell in 1960 to the spy rings of John Walker from the 1960s to 1985 to the secret document collection and publication by Edward Snowden in 2013. Kim Philby comes into the story, having managed to position himself in Washington in a job that allowed him to collect and pass on the intelligence community's most intimate secrets (including the current status of its ability to decrypt Soviet codes and the progress being made at identifying Soviet agents within the US).

The agency's commitment to the polygraph as a way of evaluating employees' loyalty is, according to Budiansky, another source of organizational failure; the polygraph had no scientific validity, and the confidence it offered permitted the agency's security infrastructure to forego other more reliable ways of combatting insider spying.

As subsequent events would make all too clear, the touching faith that a piece of Edwardian pseudoscientific electrical gadgetry could safeguard the nation’s most important secrets would prove farcically mistaken, for almost every one of the real spies to betray NSA in the ensuing years passed a polygraph interview with flying colors, while obvious signs that in retrospect should have set off alarm bells about their behavior were blithely ignored, largely due to such misplaced confidence in hocus-pocus. (kl 3355)

Budiansky makes it clear that the extreme secrecy embedded within NSA was one of the organizational and political weaknesses of the entity. Its activities were kept secret from the political authorities of the country, and the agency was sometimes used to conceal intelligence considered to be harmful to those authorities. The case of the misuse of intelligence during the Tonkin Gulf crisis is a particularly clear example, where intelligence data were misused to support the administration's need to find an incident that could serve as a cause for war.

A classified, searingly honest accounting by NSA historian Robert J. Hanyok in 2001 found that in bolstering the administration’s version of events, NSA summary reports made use of only 15 of the relevant intercepts in its files, suppressing 122 others that all flatly contradicted the now “official” version of the August 4 events. Translations were altered; in one case two unrelated messages were combined to make them appear to have been from the same message; one of the NSA summary reports that did include a mention of signals relating to a North Vietnamese salvage operation obfuscated the timing to hide the fact that one of the recovered boats was being taken under tow at the very instant it was supposedly attacking the Maddox and Turner Joy . The original Vietnamese-language version of the August 4 attack message that had triggered the Critic alert meanwhile mysteriously vanished from NSA’s files. (kl 5096)

Budiansky is forthright in identifying the weaknesses and excesses of NSA and the intelligence services. But he also makes it clear how essential these capabilities are, from allowing the US to assess Soviet intentions during the Cuban Missile crisis to directing aircraft to hostile fighters on the basis of penetration of the air-to-air radio network in Korea and Vietnam. So the hard question for Budiansky, and for us as citizens, is how to structure and constrain the collection of intelligence so that it serves the goal of defending the country against attack without deviating into administrative chaos and politicized misdirection. There are many other expert organizations that have very similar dysfunctions, from advanced civilian scientific laboratories to modern corporate IT organizations. (Here is a discussion of Paul Rabinow's ethnography of the Cetus Corporation, the biotech research firm that invented PCR; link.)