Databricks Adds ML Model Export
Databricks Adds ML Model Export
Written by Kay Ewbank   
Monday, 19 March 2018

Databricks has added a machine learning model export feature that can be used to export models from Apache Spark MLib.

 

databrickslogo

 

Apache Spark is a free community edition of the Databricks cloud-based big data platform. It is implemented in Scala and Java, runs on a cluster, and improves on Hadoop MapReduce performance, running programs up to 100 times faster in memory and ten times faster on disk, according to Apache. The Databricks commercial product is its Unified Analytics Platform. This run an optimized version of Spark that can be between 10 and 40 times faster, along with interactive notebooks, integrated workflows, and full enterprise security.

The new feature is the Databricks ML Model Export, and it can be used to export models and full machine learning pipelines from Apache Spark MLlib. These exported models and pipelines can be imported into other (Spark and non-Spark) platforms to do scoring and make predictions. This new feature is designed to provide an alternative to batch and streaming prediction within Spark. The Model Export lets you achieve very low latency in the milliseconds range, and paves the way to using ML models and pipelines in custom deployments.

MLlib models are exported as JSON files, with a format matching the Spark ML persistence format. The key changes from MLlib’s format are the use of JSON instead of Parquet, and the addition of extra metadata. This extra metadata allows scoring outside of Spark.

The list of supported models starts with full ML pipelines that contain supported transformers and models. The pipelines must be trained. Specific model types that can be exported in this release are decision tree classifier; decision tree regression; logistic regression; random forest classifier; and random forest regression. Support for more model types will be added in future releases.

 

databrickslogo
 

More Information

Databricks Website

Related Articles

Apache Spark With Structured Streaming

Spark BI Gets Fine Grain Security

Spark 2.0 Released

Apache Spark Technical Preview

Spark Announcements

Apache Releases Spark 1.6

Spark 1.4 Released

SPARQL Moves Closer

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on, Twitter, FacebookGoogle+ or Linkedin.

Banner


Is Google's Fuchsia OS Destined For Android?
20/07/2018

Insiders on the Fuchsia team, a new OS that Google is keeping under wraps, have leaked some interesting hints as to what and why it is all about. The one fact here is that there is no official word fr [ ... ]



ONVIF Challenge Winners
12/07/2018

The results of the ONVIF Open Source Spotlight Challenge have been announced. The challenge site has details of all the submissions and if you are looking for interesting ways to use security cameras  [ ... ]


More News

justjsquare

 



 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 19 March 2018 )
 
 

   
Banner
RSS feed of news items only
I Programmer News
Copyright © 2018 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.