Databricks Runtime for Machine Learning
Written by Kay Ewbank   
Monday, 15 April 2019

Databricks Runtime for Machine Learning is now generally available, offering native integration with popular ML/DL frameworks, such as scikit-learn, XGBoost, TensorFlow, PyTorch, Keras, and Horovod.

Databricks Runtime for Machine Learning is a machine learning runtime that contains multiple popular libraries, including TensorFlow, PyTorch, Keras, and XGBoost. It also supports distributed training using Horovod. The benefit of Databricks Runtime ML is that it provides a ready-to-go environment for machine learning.

databrickslogo

Databricks Runtime ML aims to be easy to use. The main libraries supported in it come pre-configured as part of the software, including HorovodRunner, which makes it possible to use the distributed deep learning framework Horovod. Horovod can be trickier to use than some frameworks because it requires you to share code and libraries across nodes, configure SSH, and execute MPI commands. HorovodRunner avoids the needs for these requirements by providing an API to allow you to use Horovod.

The team at Databricks has set up the most popular machine learning libraries as “top-tier” libraries. For these “top-tier” libraries, Databricks plans to make faster updates and provides advanced support. The top-tier libraries are:

  • TensorFlow / TensorBoard / tf.keras
  • spark-tensorflow-connector
  • PyTorch
  • Horovod / HorovodRunner
  • GraphFrames

Performance is another area to have been improved since the beta was released. In this release, improvements have been made to both Apache Spark MLlib logistic regression and tree classifiers. When running in Databricks Runtime for ML, the team at Databricks has observed around a 40% speed-up in Spark Performance Tests compared to Apache Spark 2.4.0.

The GraphFrames library in Databricks Runtime for ML also contains an optimized implementation that runs twice as fast as open-source GraphFrames and supports bigger graphs. In addition, Graph queries will make use of Spark cost-based optimization (CBO) to determine the join orders if the underlying node and edge tables contain column statistics. This can lead to as much as 100 times speed up, according to Databricks.

 

databrickslogo 

 

 

More Information

Databricks Website

Related Articles

Databricks Adds ML Model Export

 Databricks Delta Adds Faster Parquet Import

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Azure AI And Pgvector Run Generative AI Directly On Postgres
26/03/2024

It's a match made in heaven. The Azure AI extension enables the database to call into various Azure AI services like Azure OpenAI. Combined with pgvector you can go far beyond full text search. Let's  [ ... ]



Quantum Company Wins Digital Startup Contest
03/03/2024

Quantum specialists Qilimanjaro is the winner of this year's Four Years From Now competition. The award was made at this year's Mobile World Congress (MWC) in Barcelona.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 15 April 2019 )