Netflix Opensources Metaflow
Thursday, 12 December 2019

Netflix has open sourced Metaflow, which the developers describe as a human-centric framework for data science. The framework has been used internally at Netflix for the last two years to build and manage hundreds of data-science projects.

Metaflow is a Python library that provides a unified API to the infrastructure stack that is required to execute data science projects, from prototype to production. It lets data scientists structure their workflow as a Directed Acyclic Graph of steps.


Netflix developed Metaflow to boost the productivity of data scientists working on a wide variety of projects from classical statistics to state-of-the-art deep learning. The aim was to make problems in model prototypes visible earlier in the process. In February Netflix said that Metaflow had cut model deployment times from four months to seven days. 


The steps can be arbitrary Python code, with data and models stored as normal Python instance variables. They work even if the code is executed on a distributed compute platform, which Metaflow supports by default, thanks to its built-in content-addressed artifact store.

The Metaflow team says that using code is important to make the framework useful. They say that while many machine learning infrastructures rely on GUIs, configuration, DSLs, or automation, they tend to fail to scale when the problems become more complex. Instead, Metaflow relies on code. They say:

"We rely on code. General-purpose programming languages are arguably the most effective tool in building functional systems for complex use cases. The main downside of code is a steeper learning curve: for the simplest use cases, almost any other modality would provide a simpler solution. To minimize this cost, we rely on easy to learn, popular languages like Python or R."


Metaflow uses existing infrastructure when feasible. In particular, it is tightly integrated with Amazon Web Services. You can use Metaflow with a variety of machine learning or data science libraries, such as PyTorch, Tensorflow, or SciKit Learn. Metaflow allows you to write your models and business logic as idiomatic Python code with not much new to learn.

There's more information on the Metaflow website and on Github.



More Information

Metaflow Website

Metaflow On GitHub

Related Articles

Netflix Releases Polynote

Sagemaker Studio - An IDE for Machine Learning

Machine Learning With App Inventor



To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Zap A Mosquito With A Pi

and a laser of course. It isn't a new idea but at last someone has given it a go and made some results available. You might be surprised what the biggest problem is.

Linux Foundation Launches Digital Agriculture Project

The Linux Foundation has launched an open source digital infrastructure project for agriculture with the aim of promoting global collaboration across the agricultural industry. AgStack will aim to inc [ ... ]

More News





or email your comment to:


Last Updated ( Thursday, 12 December 2019 )