ModelOps is NOT a one-size-fits-all approach. It's important to identify the appropriate level of sophistication based on both the organization’s current readiness and its current and future business needs.

My previous articles on Medium talked about why organizations should choose ModelOps, and how to implement ModelOps using a holistic approach covering process, people, and technology. This article explains the levels of sophistication available when implementing ModelOps, and how to choose the right level.

Three levels of sophistication

We have identified three main levels of ModelOps sophistication. However, in practice, there is a fourth: many organizations may not yet even be at Level 1, especially if they do not use any predictive analytics. These organizations should therefore target Level 1 as a first step, even if their business needs suggest that a higher level would be appropriate. Other organizations will have higher levels of analytics skills and maturity and can start from a more advanced level if their business need demands it.

Level 1: Handcrafted

This is the first level. It is not very sophisticated and is perfect for organizations that are new to ModelOps and looking to experiment and learn before they consider including more automation in the process. It can also be the right solution when there is no real need to scale analytics. This can be because the business does not require many models or can accept longer deployment cycles. It can also be appropriate when organizations do not have the culture to support the agile methodology and effective collaboration across departments that is needed to support ModelOps.

In Level 1, every step of the process is manual, ad-hoc, and usually delivered by one team. This is generally the data science team, with very little input from the IT/Ops functions. When a deployment is required, the model is executed manually, and results are delivered manually to endpoints or users. However, this is still an effective model operationalization, because the model is being put into production at an endpoint and is working to achieve a business goal. Feedback is also provided to the model development process.

Level 2: Automated

At Level 2, automation is introduced. This level is appropriate for organizations who have some or all of the following criteria:

• Time sensitivity for responding to business conditions, which demands more efficient model deployment;
• Limited bandwidth to create, manage and deploy the models required to meet business goals; and
• A drive to challenge their culture, increase efficiency, and make changes.

In Level 2, chosen steps are automated, usually scripted, and models are delivered to an endpoint such as an internal or external application or to support a decision. Responsibilities are distributed between different teams and model governance is starting to be required to ensure models are always up-to-date.

Level 3: Industrialized

The highest level of sophistication and efficiency is Level 3, where the process is industrialized. This level applies to organizations who have at least some of the following criteria:

• A well-established DevOps culture with agile/lean methodologies;
• A large number of models, probably over 100, with defined business indicators to guide automation processes; and
• A desire to automate where necessary, understanding the need for human intervention and checks during the process.

In Level 3, ModelOps is based on a standardized and enterprise-wide process model that encourages collaboration. Proactive fault handling is key. A culture of no-blame and failing fast is supported by automated and flexible software for fast iterations of models supporting business-critical applications.

A culture of no-blame and failing fast is supported by automated and flexible software for fast iterations of models supporting business-critical applications.

Automated integration and continual model governance mean new models are developed and deployed once any model monitoring business-led KPIs are triggered.

ModelOps is not for everyone. It provides benefits of a faster time to value, better and more justifiable business outcomes, and the ability to scale analytics. However, some organizations need to increase their level of analytics maturity before they can really take advantage of a ModelOps framework. If ModelOps is the right approach, however, the level of sophistication required will depend on both the business objective and the organization’s level of analytical maturity.

If you have any questions, please post them in the comments or reach out to me directly. I would love to have a conversation about ModelOps.To discover your level of ModelOps readiness take the short Instant Insights: ModelOps Assessment or learn more on Operationalizing Analytics.