IBM Introduces Hybrid Cloud AI/ML Framework
Thursday, 15 July 2021

IBM is introducing CodeFlare, a serverless framework that aims to reduce the time and effort developers spend training and preparing AI and machine learning models for deployment in hybrid cloud environments.

CodeFlare is a framework to simplify the integration, scaling and acceleration of complex multi-step analytics and machine learning pipelines on the cloud.


CodeFlare can be used for pipeline execution and scaling with tools to define and execute parallel pipelines. The developers say it unifies pipeline workflows across multiple frameworks while providing nearly optimal scale-out parallelism on pipelined computations.

It supports a serverless user experience by integrating Red Hat OpenShift and IBM Cloud Code Engine and providing adapters and connectors for loading data and connecting to data services.

CodeFlare is built on top of Ray, an open-source distributed computing framework for machine learning applications. CodeFlare adds elements to make it easier to scale workflows, so extending the capabilities of Ray.

CodeFlare also has a Python-based interface for creating pipelines that simplifies the task of training and optimizing the model through data cleaning, feature extraction, and model optimization. The researchers say the goal of the new framework is to unify pipeline workflows across multiple platforms without requiring data scientists to learn a new workflow language.

The team says CodeFlare makes the process of analyzing and optimizing pipelines significantly faster, giving the example of one user using the framework for approximately 100,000 pipelines for training machine learning models, who found CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes.

CodeFlare has been made available open source on GitHub, and the developers have written a series of technical blog posts on how it works and what you need to get started using it. They say they've started applying this tech to things they’re building at IBM in AI research.

Future plans include support for more complex pipelines, enhanced fault-tolerance and consistency, along with improved integration and data management for external sources, and adding support for pipeline visualization.


More Information

CodeFlare On GitHub


Related Articles

IBM Releases CodeNet Dataset For AI Coding 

IBM's Elyra AI Toolkit

New MIT–IBM Watson AI Lab

Google Comics Factory Makes ML Easy

Nearly A Third Of Devs Using AI And ML

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Google Drops APKs For Android

Google announced the Android App Bundle AAB  in 2018 and it looked like a good idea - better than the APK we have all been used to. Now it has been announced that in August 2021 the APK will [ ... ]

Machine Learning Scholarship Program On Udacity

The deadline for applying for a scholarship to one of Udacity's most popular Nanodegree Programs has been extended to July 12, 2021. Apply now and gain exclusive access to a Foundations Course that co [ ... ]

More News





or email your comment to:

Last Updated ( Thursday, 15 July 2021 )