NSA Confirmed It Performed Warrantless Searches Of U.S. Citizens Phone Calls And Emails
Surprise! Metadata About Your Online Activity Reveals Where You've Been

Predicting With The Spies. The Intelligence Community Wants People Good At Predicting World Events

Good Judgment Project logo While writing today's blog post, I could have easily used, "Predicting For The Spies" instead of "Predicting With The Spies." Last week, National Public Radio (NPR) reported about the Good Judgment Project, an experiment sponsored by the U.S. intelligence community (e.g., NSA, CIA, NRO, etc.), to harness the predictive power of groups by using citizens to predict world events. NPR reported:

"According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information, and many of the people involved in the project have been astonished by its success at making accurate predictions."

The predictive power of groups is based on research that while each individual's prediction will vary greatly with error, the average prediction of the group is far more accurate. Sample questions:

"Will any country in the Euro zone default on bonds in 2014?" or "Which party will win the most seats in the next parliamentary election in Egypt?"

NPR described one citizen participant in the GJP experiment and her high success rate at predicting world events:

"She's in the top 1 percent of the 3,000 forecasters now involved in the experiment, which means she has been classified as a superforecaster, someone who is extremely accurate when predicting stuff like: Will there be a significant attack on Israeli territory before May 10, 2014?"

Three people co-lead the GJP experiment:

  • Phil Tetlock, the Leonore Annenberg University Professor in Democracy and Citizenship at the University of Pennsylvania. He is the author of the award-winning Expert Political Judgment.
  • Barb Mellers, the George Heyman University Professor at the University of Pennsylvania with appointments in the Department of Psychology and the Marketing Department of the Wharton School of Business.
  • Don Moore, an Associate Professor in the Management of Organizations group at the Haas School of Business at the University of California Berkeley. He and Max Bazerman wrote the text Judgment in Managerial Decision Making.

The GJP experiment described itself as:

"We are participating in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Activity). The ACE Program aims "to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts." The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public."

GJP participants do not have access to classified information. The GJP experiment is currently operating in season three of its four-year plan. It is not accepting any more participants for season three, which ends in May 2014. If you want to participate, you can apply online for season four, which starts in July 2014. Not all applicants are accepted. Based upon the application form, the project seems to prefer participants with degrees from accredited higher education institutions.

IARPA described three goals of its ACE program:

"The ACE Program seeks technical innovations in the following areas: (a) efficient elicitation of probabilistic judgments, including conditional probabilities for contingent events; (b) mathematical aggregation of judgments by many individuals, based on factors that may include: past performance, expertise, cognitive style, metaknowledge, and other attributes predictive of accuracy; and (c) effective representation of aggregated probabilistic forecasts and their distributions."

The NPR article asked a very relevant question:

"How is it possible that a group of average citizens doing Google searches in their suburban town homes can outpredict members of the United States intelligence community with access to classified information?"

While the researchers seem to believe that the answer is based upon the predictive power (e.g., accuracy) of a group's average prediction, I think that context matters. One must look at the broader picture for an answer.

NSA Android logo With the NSA's dragnet surveillance program, is it collecting more information than it can process? The NSA built this new $2 billion facility to store all of the data it collects. At the SXSW conference earlier this year, Snowden and other panelists discuss how mass surveillance on everyone wastes resources. When a government collects too much information or too much of the wrong information (e.g., data about innocent people; spying on mobile games; violating citizens' privacy when searching non-citizens' communications; inserting back doors inside operating system software; breaking all encryption systems; secret courts, laws, and processes), it places a priority on analyzing and sifting through the information collected (e.g., making predictions).

NSA Inside logo From the documents released since last summer, the extensive NSA surveillance seems to collect everything it can because it can, through both warrant-backed and warrantless searches where the assumption of wrongdoing is tenuous at best. A more targeted data collection means less data to analyze, less wasted resources, and an either time making predictions; or more accurate predictions. Said simply, collect less and focus your energies (and skills) at improving your predictions. Then, you wouldn't need help from a group of citizens to predict world events.

What is your opinion of the Good Judgment Project? Of the intelligence community sponsoring this experiment?


Feed You can follow this conversation by subscribing to the comment feed for this post.

Chanson de Roland

While no one knows why the average of a large group of people, who use only open-source intelligence, is better at predicting world events, statistics provides an answer, which is not complete satisfying but which is probably true: The wisdom of a sufficiently large crowd, that is, a crowd of adequate sample size for the level of confidence, corrects for error by random sampling, so that what is left is the average, which is the best true estimate of human judgment.

In addition, as Mr. Jenkins writes, supra, the humans in the sample don't try to collect all information, much less analyze all information, but use various strategies for focusing on what they judge to be relevant information. Some of those collective strategies will be great, others good, and some awful, but random sampling will find the best estimate of all those strategies, with the errors canceling each other out, as Professor Tetlock explains in the article.

So the problem for the CIA et al. is that even if they opt to use various strategies to identifying and analyzing relevant information, it will be limited by its number of analysts. What CIA may do is use the best strategies from GJP and ACE to enhance their predictive success.

However, whether any of those strategies is amenable to being reduced to an algorithm or whether the CIA must train its analysts to use them will be interesting. But using keywords and pattern recognition techniques to scan everything is of dubious value, especially since our adversaries know that we are doing it. So it is well past time to enhance our predictions by using the best strategies for identifying and analyzing relevant information and/or the collective judgments from sufficiently large samples of people, who use open-source intelligence and their various strategies and intuitions to make predictions.

The comments to this entry are closed.