"At a time when our need for policy agility has never been greater, our traditional 20th century tools for tracking international development cannot keep up. Too often, by the time we have evidence of what is happening at the household level, the harm has already been done … And the irony is that we are actually swimming in an ocean of real-time information.”

With that statement in a November 2011 address, United Nations Secretary General Ban Ki-moon declared the need for more sophisticated methods to understand, in near real-time, what is happening in the world. United Nations Global Pulse was created to research ways analytics can make that possible.

The world has now reached the point where billions of digital comments are posted daily on public web forums, blogs, and social media sites like Facebook and Twitter. These public comments express sentiments about employment, transportation, cost of living, and anything else that’s on people’s minds. If we collect and analyze all of these public comments with respect to current events (financial crisis, natural disasters, sporting events), clear and common sentiments begin to emerge.

For example, a recent study on unemployment conducted by SAS and the UN Global Pulse (@unglobalpulse) revealed that increased chatter about postponing vacations, increasing use of public transportation and downgrading one’s automobile could, indeed, predict an unemployment spike in the US and Ireland. While these relationships are not surprising, the analytics quantified the amount of time these conversations usually precede an event (i.e. chatter about delaying travel typically preceded unemployment spikes by five months in the US).

Could social media analysis replace the need for official statistics? Not in my opinion. Social media analysis is an excellent complement to official statistics, often bringing more insights. For example, official statistics may tell us that there is a decline in the number of children enrolled in school in a developing region, and the public chatter may tell us why children are being pulled out of schools (to work in the local markets for food, to protect the village from civil unrest, etc.) And, as shown in the project with UN Global Pulse, the analysis can also determine the types of chatter that usually precede or follow an event captured by official statistics.

I believe this is just the tip of the iceberg of what we can do with social media and other non-traditional big data sources. For example, what if we took anonymized credit card data on spending habits or anonymized mobile phone data to see how people are moving before and after a crisis event? Similar to how social media analysis enriches official statistics, these other data sources would also provide additional detail and perspectives to tell a clearer story. Or how about also using satellite data to tell us how quickly things are growing throughout a country experiencing drought conditions, as a way to help understand migration patterns? The possibilities are endless. However, it all comes back to the UN Secretary General’s belief, that big data is useless without analytics.