Operationalization of Machine Learning (ML) models looks set to be the main challenge of 2019. At the end of 2018 up to 85% of all machine learning models had still not yet been used in production.
Getting your ML models into production reliably, quickly and safely is no mean feat. We’ve talked to hundreds of teams and discovered that organizations fall into three stages of maturity:
Level 1: Playing with AI
The organization has a stated AI strategy, most likely this has come from the C-suite. Current progress on this is that there are a number of PoC projects within the organization with the intention of proving that AI can be applied to business problems. There may be a data science team which is small or it may be outsourced to a data science consultancy. Work is collaborated on manually, perhaps tracked in a spreadsheet. A number of models have been developed which show the potential to be useful in production but these have not been deployed yet.
Level 2: Initial Deployment
These organizations are deploying models into production however it’s a largely uncontrolled process with little automation, monitoring or governance. Development of models is slow and manual, with increasing inefficiency as teams grow. Data Scientists may be treading on each others’ toes and deployment of models is slow and error prone. There is no reliable tracking of data, code and model provenance and monitoring of models in production is insufficient to identify problems with decision-making.
Level 3: Operationalized AI
This stage of maturity is often seen in organizations where AI is central to competitive advantage, or in regulated industries. Here the AI is fully reproducible and its decisions explainable. Audits are easily supported due to comprehensive provenance of development of models. Models are monitored throughout their lifecycle, so that problems with performance can be uncovered and fixed quickly. Some form of MLOps or DevOps for ML is in place to allow fast, safe deployment of models. Teams are able to collaborate using distributed version control of their whole experiment which supports safe scaling and reduces human error.
To learn more about where your organization is on this scale, and to get tailored advice on how to move to Level 3, we offer an AI Maturity certification which gives you a clear assessment of your current state and key steps to move forward safely.