I recently presented a session on big data at the 13th Annual Privacy and Security hosted by the Province of British Columbia and held in Victoria. There were a number of interesting discussions and presentations that relate to privacy and security ramifications of big data. The discussion was timely given the recent news about privacy and big data implementations, including coverage by the New York Times. Although I understand the desire to leverage a hot topic like big data to drive interest, I feel most of the security and privacy considerations are more appropriately driven by the business discipline that is being utilized. For example, privacy and security policies and approaches related to “one to one” marketing or personalization are not a big data issue, they are a customer management issue. In this context, striking the correct balance between leveraging personal information to provide better service vs. being too invasive needs to be hammered out by marketing, legal, IT, etc. While the approach needs to be extended to accommodate big data, it’s not a big data security issue per se.

Although the conference was not focused on big data, there was plenty of interest and discussion about the impact and potential for big data, most in the context of what it can do for government.

• Dr. Margaret MacDiarmid, Minister of Labour, Citizens’ Services and Open Government, spoke about how they have made 2,400 datasets available to the public as part of their open government policies.
• Cory Doctorow, editor of BoingBoing, spoke about the privacy bargain, the notion of providing personal information for better service. He explained that “we are bad at determining the future impact of providing personal information now” and noted that the “privacy bargain should not be based on people not knowing how to use their computer”.

My session addressed the potential impact of big data in the government sector and how big data can help drive improvements in Public Health, Service & Public Trust, Innovation & Efficiency, and Open Government.

I addressed several considerations for success in big data, including:

• Starting with ensuring common goals and alignment across the organization about the impact of analytics
• Extending your enterprise architecture approach to accommodate big data, vs. defining an independent big data strategy
• Leveraging a strategic information management approach, and extending that discipline to big data
• Devising a resource plan that will ensure that you have the necessary analytic skills such as data scientists, educating IT on the analytics lifecycle, etc.
• Extending your governance, security, and privacy efforts to incorporate big data.

Although time was limited, I also introduced a framework or set of considerations relating to privacy and security.  The overall message was that they shouldn’t devise a separate security approach for big data, they should extend or augment what they already have in place so that big data is a natural part of their overall security plan. We discussed that security should be comprehensive, and that big data considerations should be factored into the design, implementation and monitoring aspects of their security framework.

And no surprise that there were several questions about Hadoop. We discussed that since the Hadoop security features are not well established or completely hardened at this point, that special considerations need to be put in place for data that is stored, processed and analyzed in Hadoop. But we’ll have to defer a more specific discussion around Hadoop for a later post.