Databricks Adds ML Model Export
Written by Kay Ewbank   
Monday, 19 March 2018

Databricks has added a machine learning model export feature that can be used to export models from Apache Spark MLib.

 

databrickslogo

 

Apache Spark is a free community edition of the Databricks cloud-based big data platform. It is implemented in Scala and Java, runs on a cluster, and improves on Hadoop MapReduce performance, running programs up to 100 times faster in memory and ten times faster on disk, according to Apache. The Databricks commercial product is its Unified Analytics Platform. This run an optimized version of Spark that can be between 10 and 40 times faster, along with interactive notebooks, integrated workflows, and full enterprise security.

The new feature is the Databricks ML Model Export, and it can be used to export models and full machine learning pipelines from Apache Spark MLlib. These exported models and pipelines can be imported into other (Spark and non-Spark) platforms to do scoring and make predictions. This new feature is designed to provide an alternative to batch and streaming prediction within Spark. The Model Export lets you achieve very low latency in the milliseconds range, and paves the way to using ML models and pipelines in custom deployments.

MLlib models are exported as JSON files, with a format matching the Spark ML persistence format. The key changes from MLlib’s format are the use of JSON instead of Parquet, and the addition of extra metadata. This extra metadata allows scoring outside of Spark.

The list of supported models starts with full ML pipelines that contain supported transformers and models. The pipelines must be trained. Specific model types that can be exported in this release are decision tree classifier; decision tree regression; logistic regression; random forest classifier; and random forest regression. Support for more model types will be added in future releases.

 

databrickslogo
 

More Information

Databricks Website

Related Articles

Apache Spark With Structured Streaming

Spark BI Gets Fine Grain Security

Spark 2.0 Released

Apache Spark Technical Preview

Spark Announcements

Apache Releases Spark 1.6

Spark 1.4 Released

SPARQL Moves Closer

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Ibis 8 Adds Streaming
05/03/2024

Ibis 8.0 has been released with stream processing backends. The new release includes Apache Flink as a streaming backend, and RisingWave, a streaming database backend. There's also a new batch backend [ ... ]



WasmCon 2023 Sessions Now Online
01/03/2024

The recorded session of the premier conference for technical developers and users interested in exploring the potential of WebAssembly are now online.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 19 March 2018 )