Databricks Adds ML Model Export
Databricks Adds ML Model Export
Written by Kay Ewbank   
Monday, 19 March 2018

Databricks has added a machine learning model export feature that can be used to export models from Apache Spark MLib.

 

databrickslogo

 

Apache Spark is a free community edition of the Databricks cloud-based big data platform. It is implemented in Scala and Java, runs on a cluster, and improves on Hadoop MapReduce performance, running programs up to 100 times faster in memory and ten times faster on disk, according to Apache. The Databricks commercial product is its Unified Analytics Platform. This run an optimized version of Spark that can be between 10 and 40 times faster, along with interactive notebooks, integrated workflows, and full enterprise security.

The new feature is the Databricks ML Model Export, and it can be used to export models and full machine learning pipelines from Apache Spark MLlib. These exported models and pipelines can be imported into other (Spark and non-Spark) platforms to do scoring and make predictions. This new feature is designed to provide an alternative to batch and streaming prediction within Spark. The Model Export lets you achieve very low latency in the milliseconds range, and paves the way to using ML models and pipelines in custom deployments.

MLlib models are exported as JSON files, with a format matching the Spark ML persistence format. The key changes from MLlib’s format are the use of JSON instead of Parquet, and the addition of extra metadata. This extra metadata allows scoring outside of Spark.

The list of supported models starts with full ML pipelines that contain supported transformers and models. The pipelines must be trained. Specific model types that can be exported in this release are decision tree classifier; decision tree regression; logistic regression; random forest classifier; and random forest regression. Support for more model types will be added in future releases.

 

databrickslogo
 

More Information

Databricks Website

Related Articles

Apache Spark With Structured Streaming

Spark BI Gets Fine Grain Security

Spark 2.0 Released

Apache Spark Technical Preview

Spark Announcements

Apache Releases Spark 1.6

Spark 1.4 Released

SPARQL Moves Closer

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

Banner


GitLab Adds Auto DevOps
26/06/2018

GitLab 11.0 has been released with support for Auto DevOps and License Management. The open source GIT based repository management software system first announced Auto DevOps last year.



GitHub Enterprise 2.14 Adds Unified Search
13/07/2018

GitHub Enterprise has been updated to add unified search and a beta of the Checks API.


More News

justjsquare

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 19 March 2018 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2018 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.