Data science is heading for a problem. We know that it is a "shortage speciality" – that demand for data science skills outweighs the supply. However, a survey by McKinsey two years ago shows that the situation could be even worse than expected.
Around one-third of those responding to McKinsey’s survey said that they believed that automation would result in significant skills shortages in data analytics within the next three years. Worse, those making this observation were the companies with more experience of automation and AI use. The general consensus was that the companies that automated first had their pick of data scientists, and therefore avoided skill shortages. This study shows that assumption was wrong.
Skills shortages in data science are likely to affect any and every company and organisation that tries to automate and adopt AI over the next few years. It was 2012 when Harvard Business Review dubbed data science the sexiest job of the 21st century – but the mismatch between supply and demand has not improved since then. Something urgently needs to change.
Skills shortages in data science are likely to affect any and every company and organisation that tries to automate and adopt AI over the next few years.
From continuous improvement to continuous adaptation
We know and understand that automation and AI are changing the business landscape. However, we have to recognise that the education landscape also needs to change to match. We are moving from the continuous improvement of the 1990s to a focus on continuous adaptation instead. Industries need people with the right skills, and particularly an understanding of data. And universities need to deliver graduates with those skills. Furthermore, things are changing rapidly, all the time. The bottom line is that the best way to match those needs is through collaboration.
What skills are most in demand just now? Analysis by LinkedIn suggests that both hard (technical) and soft (interpersonal) skills are required. The top five soft skills include creativity, persuasion, collaboration, adaptability and emotional intelligence. Technical skills include analytical reasoning, artificial intelligence, user experience design and business analysis. These all seem reasonable areas for universities and businesses to focus on together.
However, there are conflicting paradigms between universities and businesses. Universities are largely funded by governments, albeit sometimes indirectly, and have a strong research focus. Their emphasis is on open access to information and scientific inquiry. Businesses, however, want to keep commercial information secret. They need value for shareholders, and they focus on problem solving and innovation. What happens, then, when businesses provide funding for universities as part of collaborative activities? Who gives way?
Changing the landscape
There is a growing sense that the traditional university model of learning for learning’s sake may not be sustainable in the future. The changing world of work means that employers want work-ready graduates – and graduates want jobs. There is increasing international competition for students and their fees. Nontraditional education rivals are offering new alternatives to traditional degrees, and if they offer better value for money, they are being adopted. Education is increasingly seen as a continuous process. There is a growing challenge to whether a three- or four-year degree really fits the needs of the modern world, and universities are coming under increasing pressure to change.
Many universities are already adapting. The more entrepreneurial among them are becoming more "learner-centric" and starting to work much more closely with businesses. Many are also moving into a more virtual world – a move that is being accelerated by the new coronavirus. They are setting up as centres of excellence, with industry funding. They are seeking teaching from those with industrial experience and moving to more industry-focused research. The buzzword is now co-creation, far from the old image of ivory towers.
A framework for the future
These entrepreneurial universities are starting to offer very different data science courses. They recognise that technical skills are important, but that people also need to be taught soft skills. A framework for data science teaching for the future must combine the two through a curriculum co-created with business.
The basic building blocks from among hard skills include mathematical and statistical skills, numerical method programming and data curation skills. Subject matter and domain knowledge are also important, as are project management and creative problem solving – which, though technical, could also be considered soft skills in many ways. On the soft skills side, the ideal framework would blend emotional intelligence with intellectual ability, adaptability and ability to get things done. This type of framework is likely to deliver the closest approximation possible to a work-ready data scientist.
