Last week, I reflected on how much had changed since I joined SAS 20 years ago, as a fresh-faced 23 year old. It is tempting to say that everything is different, but it really would not be true. There are many things that have stayed the same, even in the fast-moving and rapidly-changing world of analytics. New graduates please note…
Observational data is a pain in the neck
Without data, analytics does not exist. And particularly, without data that accurately reflects the state of something or someone, analytics will not provide insight and support better decision-making. Getting the right data, however, is always a problem. It always was, and this is not going to change.
Data pre-processing takes too much time
Personally, I have never believed that 80% of an analyst’s time is spent on data preparation. It is, however, certainly true that a large proportion of analytical effort is spent on straightening out the data, and understanding how it has been collected and can be used. Some of the newer data sources such as language, images and video are particularly ‘dirty’ in their raw state, so the time required for data preparation will only increase, unless we find ways to automate the process.
Statistical theory underpins even machine learning algorithms
A regression model or a multi-layer perception neural network, whether from the 1950s or now is still a linear regression or a multi-layer perception neural network. The theory has not changed, although it is now possible to use these tools on larger volumes of data, allowing us to solve different types of problems.
Explaining, interpreting and communicating results to stakeholders is key
I have yet to find anyone who is prepared to use analytics in their business if they do not understand it or cannot see its value. As an analyst, you have to be able to “sell” the concept and build empathy with stakeholders. Otherwise analytics—and the emerging insights—will not be adopted and your time will have been wasted.
You need to start with a definition of the business problem
During the last 20 years, I have seen many organisations spending too much time looking at the newest data form or technology and forgetting the business problem they are trying to solve. If solving your problem will not have a significant impact, then it is not worth bothering. Many companies, vendors and individuals have fallen into this trap, and one part of an analyst’s role is helping customers to define the problem clearly, not just looking for a way to build a cool model.
The end-to-end process of gathering data, and discovering and deploying insights remains poorly understood
Many organisations still seem to believe that you can just throw data into a magic melting pot to get a revolutionary result powered by machine learning and/or artificial intelligence. The concept of an analytics lifecycle has been around for at least 20 years and remains relevant, but the key is how quickly we can move through it. Data scientists need to be a lot more open about explaining their models, and increasing understanding that there is no magic involved.
Companies struggle with the cultural change that comes with analytics
Changing the culture of an organisation takes time, and is hard work. Industries that are either regulated or discovered the value of analytics early have often not evolved their use of analytics much further. Many have struggled to change their organisational culture to adapt to new ways of working with analytics, and are therefore not getting full value from analytics. Innovation in these industries comes from either new entrants or organisations with dynamic leaders who live, breathe and install analytics at the heart of their organisation.
People still find it hard to understand that visualisation is not the only form of analytics
Almost everyone understands traditional analytical visualisations such as a histograms, scatter plots or pie charts. This is great but there are major weaknesses in using these methods. In particular, their interpretation is susceptible to human bias, and they are therefore often used to support a theory rather than to discover new insights. Newer visualisations, underpinned by advanced analytical techniques such as decision trees, network diagrams and clustering, are less prone to human bias and enable insight to be discovered quickly.
A data scientist is never satisfied
Data scientists have to constantly push the boundaries to solve problems. They are therefore never satisfied with what is available, whether data, infrastructure or analytics technology. They will always want more detailed data, delivered faster, access to the latest algorithms, to run faster on greater volumes or data. They will want to automate mundane tasks and deploy their results quicker. They are always looking for newer, better ways of doing things.
Changes upon changes
Many things have therefore been constants in my working life. Other things have changed, but have now come full circle. For example, when I started work, a young stats geek could get away with wearing jeans, jumper and trainers. In the nineties, I was expected to dress in traditional business attire—suit, tie, polished black shoes and so on. Now, however, we are back to a more relaxed mode.
No matter what changes we see, analytics technology needs to fit into an environment where automation, reusability, governance and productivity matter. It must help organisations to make better decisions faster. No-one knows what changes the next 20 years will bring, but I am willing to bet that this, at least, will remain constant.
No matter what changes we see, analytics technology needs to fit into an environment where automation, reusability, governance and productivity matter. Learn more about our AI journey with SAS.
Thanks to Tuba, Matthew, Jeoren, Simon, Rens, Gerhard, Wayne, Andy, Pete, Lee, Jennifer, Cristina, John, Colin, Sara, Pete, Peter, Angus, Vish, Puni, Kat and Nicola for giving me the inspiration to write these blogs and to all the customers and work colleagues that I have had the pleasure of working with. It’s been emotional.
