IBM Introduces Hybrid Cloud AI/ML Framework
Thursday, 15 July 2021

IBM is introducing CodeFlare, a serverless framework that aims to reduce the time and effort developers spend training and preparing AI and machine learning models for deployment in hybrid cloud environments.

CodeFlare is a framework to simplify the integration, scaling and acceleration of complex multi-step analytics and machine learning pipelines on the cloud.


CodeFlare can be used for pipeline execution and scaling with tools to define and execute parallel pipelines. The developers say it unifies pipeline workflows across multiple frameworks while providing nearly optimal scale-out parallelism on pipelined computations.

It supports a serverless user experience by integrating Red Hat OpenShift and IBM Cloud Code Engine and providing adapters and connectors for loading data and connecting to data services.

CodeFlare is built on top of Ray, an open-source distributed computing framework for machine learning applications. CodeFlare adds elements to make it easier to scale workflows, so extending the capabilities of Ray.

CodeFlare also has a Python-based interface for creating pipelines that simplifies the task of training and optimizing the model through data cleaning, feature extraction, and model optimization. The researchers say the goal of the new framework is to unify pipeline workflows across multiple platforms without requiring data scientists to learn a new workflow language.

The team says CodeFlare makes the process of analyzing and optimizing pipelines significantly faster, giving the example of one user using the framework for approximately 100,000 pipelines for training machine learning models, who found CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes.

CodeFlare has been made available open source on GitHub, and the developers have written a series of technical blog posts on how it works and what you need to get started using it. They say they've started applying this tech to things they’re building at IBM in AI research.

Future plans include support for more complex pipelines, enhanced fault-tolerance and consistency, along with improved integration and data management for external sources, and adding support for pipeline visualization.


More Information

CodeFlare On GitHub


Related Articles

IBM Releases CodeNet Dataset For AI Coding 

IBM's Elyra AI Toolkit

New MIT–IBM Watson AI Lab

Google Comics Factory Makes ML Easy

Nearly A Third Of Devs Using AI And ML

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Conan C/C++ Package Manager 2.0 Now In Beta

A pre-release beta of the Conan C/C++ Package Manager has been released on GitHub with new generators and a new graph model that provides better support for C and C++ binaries relationships, compilati [ ... ]

Microsoft Graph Developer Proxy In Preview

Microsoft has updated the Graph Developer Proxy, which it says can be used to build resilient and performant apps. The updated version adds support for simulating errors on Microsoft Graph and other A [ ... ]

More News





or email your comment to:

Last Updated ( Thursday, 15 July 2021 )