While writing today's blog post, I could have easily used, "Predicting For The Spies" instead of "Predicting With The Spies." Last week, National Public Radio (NPR) reported about the Good Judgment Project, an experiment sponsored by the U.S. intelligence community (e.g., NSA, CIA, NRO, etc.), to harness the predictive power of groups by using citizens to predict world events. NPR reported:
"According to one report, the predictions made by the Good Judgment Project are often better even than intelligence analysts with access to classified information, and many of the people involved in the project have been astonished by its success at making accurate predictions."
The predictive power of groups is based on research that while each individual's prediction will vary greatly with error, the average prediction of the group is far more accurate. Sample questions:
"Will any country in the Euro zone default on bonds in 2014?" or "Which party will win the most seats in the next parliamentary election in Egypt?"
NPR described one citizen participant in the GJP experiment and her high success rate at predicting world events:
"She's in the top 1 percent of the 3,000 forecasters now involved in the experiment, which means she has been classified as a superforecaster, someone who is extremely accurate when predicting stuff like: Will there be a significant attack on Israeli territory before May 10, 2014?"
Three people co-lead the GJP experiment:
- Phil Tetlock, the Leonore Annenberg University Professor in Democracy and Citizenship at the University of Pennsylvania. He is the author of the award-winning Expert Political Judgment.
- Barb Mellers, the George Heyman University Professor at the University of Pennsylvania with appointments in the Department of Psychology and the Marketing Department of the Wharton School of Business.
- Don Moore, an Associate Professor in the Management of Organizations group at the Haas School of Business at the University of California Berkeley. He and Max Bazerman wrote the text Judgment in Managerial Decision Making.
The GJP experiment described itself as:
"We are participating in the Aggregative Contingent Estimation (ACE) Program, sponsored by IARPA (the U.S. Intelligence Advanced Research Projects Activity). The ACE Program aims "to dramatically enhance the accuracy, precision, and timeliness of forecasts for a broad range of event types, through the development of advanced techniques that elicit, weight, and combine the judgments of many intelligence analysts." The project is unclassified: our results will be published in traditional scholarly and scientific journals, and will be available to the general public."
GJP participants do not have access to classified information. The GJP experiment is currently operating in season three of its four-year plan. It is not accepting any more participants for season three, which ends in May 2014. If you want to participate, you can apply online for season four, which starts in July 2014. Not all applicants are accepted. Based upon the application form, the project seems to prefer participants with degrees from accredited higher education institutions.
IARPA described three goals of its ACE program:
"The ACE Program seeks technical innovations in the following areas: (a) efficient elicitation of probabilistic judgments, including conditional probabilities for contingent events; (b) mathematical aggregation of judgments by many individuals, based on factors that may include: past performance, expertise, cognitive style, metaknowledge, and other attributes predictive of accuracy; and (c) effective representation of aggregated probabilistic forecasts and their distributions."
The NPR article asked a very relevant question:
"How is it possible that a group of average citizens doing Google searches in their suburban town homes can outpredict members of the United States intelligence community with access to classified information?"
While the researchers seem to believe that the answer is based upon the predictive power (e.g., accuracy) of a group's average prediction, I think that context matters. One must look at the broader picture for an answer.
With the NSA's dragnet surveillance program, is it collecting more information than it can process? The NSA built this new $2 billion facility to store all of the data it collects. At the SXSW conference earlier this year, Snowden and other panelists discuss how mass surveillance on everyone wastes resources. When a government collects too much information or too much of the wrong information (e.g., data about innocent people; spying on mobile games; violating citizens' privacy when searching non-citizens' communications; inserting back doors inside operating system software; breaking all encryption systems; secret courts, laws, and processes), it places a priority on analyzing and sifting through the information collected (e.g., making predictions).
From the documents released since last summer, the extensive NSA surveillance seems to collect everything it can because it can, through both warrant-backed and warrantless searches where the assumption of wrongdoing is tenuous at best. A more targeted data collection means less data to analyze, less wasted resources, and an either time making predictions; or more accurate predictions. Said simply, collect less and focus your energies (and skills) at improving your predictions. Then, you wouldn't need help from a group of citizens to predict world events.
What is your opinion of the Good Judgment Project? Of the intelligence community sponsoring this experiment?