National Crowdsourcing Effort Proves the Value of Sending a Clear Message

'Grand Challenges' experiment shows social media prompts can be misinterpreted, amplified and taken too literally.

by / May 20, 2010

A national effort to elicit "grand" science and technology ideas has taught a nonprofit White House partner some lessons in crowdsourcing -- namely streamline the message.

"The prompts used to drive a crowdsourcing initiative are perhaps the most important part of the effort," an Expert Labs blog post read. "If there is an area for improvement in our efforts, this is clearly an important one to focus on."

The finding could prove useful to other government agencies that are thinking about crowdsourcing questions to the general population.

The White House Office of Science and Technology Policy (OSTP) and the National Economic Council released a request for information in February to collect public input regarding the "grand challenges" identified in President Barack Obama's innovation strategy, along with other ideas and potential partners. The mid-April deadline passed and thousands of replies were garnered.

The OSTP partnered with Expert Labs -- a nonprofit project of the American Association for the Advancement of Science -- to gather responses, which were mainly prompted by Twitter and Facebook messages. The ThinkTank app was used to capture and organize these responses, which are available in a Google document. It was the first big crowdsourcing experiment Expert Labs took on.

Some of those prompts included, "What's the next moon shot or the next human genome sequencing?" "What are the ambitious goals that are going to generate jobs, improve security, drive innovation or inspire students to learn?" and "How do we get our best minds to work on solving these challenges?"

That said, the White House prompted its social network connections to respond in several different ways, the blog post said. "Messages varied in tone, timing and in how much background expository information was provided," it said. "Clearly the Grand Challenges initiative itself was an ambitious one."

Specifically the prompts broadcast via Twitter and Facebook addressed a large population of users -- who aren't necessarily science and technology experts -- with a fairly complicated question. And even those in the science and tech communities, who may have had responses, had to adapt to the idea of sharing such ideas using what have become common social networking tools, according to the blog post.

Perhaps even more significant is the social network environment. "... The terse wording and distracted attention environment of social networks can amplify ambiguities in a prompt," the blog said.

An example of this issue related to the Grand Challenges project is the initial prompt from the White House's Twitter feed -- "The next Apollo program or Human Genome Project?" Because of this, a significant number of responses took the question very literally and answered with one of the programs prioritized over the other.

While the refining the message may be the greatest lesson learned, there are a few other key findings and highlights of the crowdsourcing effort:

  • More than 2,000 replies to the request for information were received via Twitter and Facebook within the approximately 48 hours that those mediums were available (Expert Labs announced it would use those tools about two days before the April 15 deadline).
  • The initial prompts from the White House's social network accounts (such as @whitehouse on Twitter) were forwarded by enough people to nearly double the number of people who saw the prompts.
  • Off-topic and on-topic responses can coexist on the network. One of the biggest fears of large-scale crowdsourcing is that "noise" will crowd out
  • high-quality responses. "... There are obviously responses that are frivolous, or ones that are advancing political concerns to the White House that do not relate to science or technology, and the presence of these submissions did not prevent interested parties from also suggesting ideas that may be of value," the blog post said.
  • It's possible to reliably capture a large volume of responses across multiple social networks. For example, the ThinkTank app, created by the open source community, was able to connect to the appropriate networks, collect all replies submitted by those connected to the White House's social network accounts and store them in its database system "with perfect fidelity immediately after they were submitted."
  • Amplifying a crowdsourcing prompt is as common a behavior as actually replying to it. A cursory examination of replies sent to the White House's prompts regarding Grand Challenges showed a significant number of "retweets" or forwards, or restatements of the original prompt. "While some of this is to be expected, as organizations use the networks to relay the message to their followers or members, it would also appear that some ... members see the amplification of the message as a form of participation in itself," the blog post said. "This suggests that it may be valuable to reward amplification of a prompt as well as direct responses to a prompt."

As far as next steps, the findings of the Grand Challenges experiment have been shared at the American Association for the Advancement of Science Forum on Science and Technology Policy in Washington, D.C., Expert Labs will also be asking people to analyze and present the data collected and help turn those analysis and visualization tools into parts of the ThinkTank platform.

"We're eager for our community to analyze this data, create visualizations from it and in general, provide any insights into the responses that can be gleaned," the blog post said.


Karen Wilkinson

Karen is a former staff writer for Government Technology magazine.