Netflix Opensources Metaflow
Thursday, 12 December 2019

Netflix has open sourced Metaflow, which the developers describe as a human-centric framework for data science. The framework has been used internally at Netflix for the last two years to build and manage hundreds of data-science projects.

Metaflow is a Python library that provides a unified API to the infrastructure stack that is required to execute data science projects, from prototype to production. It lets data scientists structure their workflow as a Directed Acyclic Graph of steps.

metaflow

Netflix developed Metaflow to boost the productivity of data scientists working on a wide variety of projects from classical statistics to state-of-the-art deep learning. The aim was to make problems in model prototypes visible earlier in the process. In February Netflix said that Metaflow had cut model deployment times from four months to seven days. 

 

The steps can be arbitrary Python code, with data and models stored as normal Python instance variables. They work even if the code is executed on a distributed compute platform, which Metaflow supports by default, thanks to its built-in content-addressed artifact store.

The Metaflow team says that using code is important to make the framework useful. They say that while many machine learning infrastructures rely on GUIs, configuration, DSLs, or automation, they tend to fail to scale when the problems become more complex. Instead, Metaflow relies on code. They say:

"We rely on code. General-purpose programming languages are arguably the most effective tool in building functional systems for complex use cases. The main downside of code is a steeper learning curve: for the simplest use cases, almost any other modality would provide a simpler solution. To minimize this cost, we rely on easy to learn, popular languages like Python or R."

 

Metaflow uses existing infrastructure when feasible. In particular, it is tightly integrated with Amazon Web Services. You can use Metaflow with a variety of machine learning or data science libraries, such as PyTorch, Tensorflow, or SciKit Learn. Metaflow allows you to write your models and business logic as idiomatic Python code with not much new to learn.

There's more information on the Metaflow website and on Github.

metaflow

 

More Information

Metaflow Website

Metaflow On GitHub

Related Articles

Netflix Releases Polynote

Sagemaker Studio - An IDE for Machine Learning

Machine Learning With App Inventor

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Apache Shiro 2.0 Released
21/03/2024

Apache Shiro 2.0 has been released. The Java security framework now requires at least Java 11, and has added support for Jakarta EE 10.



The University of Tübingen's Self-Driving Cars Course
22/03/2024

The recorded lectures and the written material of a course on Self-Driving Cars at the University of Tübingen have been made available for free. It's a first class opportunity to learn the in an [ ... ]


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

 

Last Updated ( Thursday, 12 December 2019 )