Machine learning is moving into the mainstream. Once the sole purview of academic researchers and advanced technology firms, machine learning is now being is used by many companies in more traditional industry verticals.

Machine learning uses mathematical (not necessarily statistical) models to learn about data. In this context, learning about data basically means curve-fitting or hyper-plane fitting, without the traditional statistical concerns about underlying distributions of the data or possible differences between the current sample of data and other samples of data. Machine learning techniques handle these concerns in different ways.

Perhaps surprisingly, such machine learning models can be quite adept at making accurate predictions about new samples of data, even when the phenomenon they are trying to predict is very rare or when it exhibits nonlinear behavior. 

Due to this ability to make such accurate predictions, industries like finance, retail, transportation and security are embracing machine learning techniques. When should you consider trying out machine learning?

The primary consideration when comparing a machine learning approach to a more traditional regression approach is the difficulty in interpreting most machine learning models. So, machine learning is often employed in situations where predictive accuracy is more important than model interpretation.

Other instances when you might consider using machine learning in place of more traditional techniques include when your data sources contain more variables than observations, many correlated variables, unstructured data, or rare or nonlinear phenomena.

Predictive modeling using machine learning is a very different process from inferring knowledge from a regression model. Practitioners may find it unsettling to switch from a mindset that values assumption-checking, parsimony, and interpretation to a mindset that values predictive accuracy over all else.

My advice is: take the plunge. Don’t try to understand the incomprehensible inner-workings of a neural network, just make sure you are not overfitting your training data. Don’t labor over every branch of a decision tree. Train a lot of decision trees and use them all in an ensemble. I could continue, but I’ll never explain the differences between machine learning and statistics, or justify machine learning approaches, as elegantly as Leo Breiman did in his seminal 2001 paper: “Statistical Modeling: The Two Cultures” If you are interested in understanding the basic ideas and philosophy of machine learning, you need to read this paper.

If you want to hear me discuss the topic in more detail, watch the webinar, Machine Learning: Principles and Practice, where I review some of the fundamental ideas of machine learning and a few lessons learned from helping customers use machine learning in the real world.

Subsequent webinars will present machine learning techniques like principal component analysis, clustering, and ensemble modeling.