Apache Ignite Adds Spark DataFrames Support
Written by Kay Ewbank   
Monday, 29 March 2021

Apache Ignite, a distributed database for high-performance computing with in-memory speed, has been updated with support for Spark DataFrames and machine learning.

Ignite can be used as a traditional SQL database via JDBC drivers, ODBC drivers, or its own native SQL APIs. By default, it runs purely in-memory, but clusters can be configured to run on a mix of disk and memory.

ignite

It supports 'co--located compute in Java, Scala, Kotlin, C#, and C++, meaning that you can create the equivalent of stored procedures in modern JVM languages, C# or C++ to develop and execute custom tasks across a distributed database. Ignite can operate in a strongly consistent mode that provides full support for distributed ACID transactions across multiple cluster nodes, caches, tables, and partitions.

Ignite supports continuous queries, meaning that rather than using a trigger to react to specific events, in Ignite you can write and run continuous queries written in languages such as Java or C# that process streams of changes on the database and application side.

Highlights of the latest version of the distributed database are general availability of its machine learning feature, and support for Spark DataFrames.

Ignite machine learning has TensorFlow integration as well as built-in algorithms and tools so it can be used to build scalable machine learning models without having to transfer data out of Ignite. You can train, deploy, evaluate, and update your ML and DL models continuously and at scale.

Perhaps even more strikingly, the latest version adds support for Spark DataFrames. The announcement emphasizes that this isn't a joke or misprint, saying that many early Ignite adopters have been hoping for this for years. A Spark DataFrame is a distributed collection of data organized into named columns, and means Spark can use the Catalyst query optimizer to produce more efficient query execution plans.

The developers say that Ignite expands DataFrame, simplifying development and improving data access times whenever Ignite is used as memory-centric storage for Spark. The support means Ignite users can share data and state across Spark jobs by writing and reading DataFrames to and from Ignite; and can improve the performance of SparkSQL queries by optimizing Spark query execution plans with the Ignite SQL engine which include​s advanced indexing. 

 ignite

More Information

Apache Ignite Website

Related Articles

Spark 3 Improves Python and SQL Support

Apache Superset Reaches Top Level Project Status

Apache Daffodil Now Top Level Project

Facebook Apollo NoSQL Database  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


TypeScript Improves Never-Initialized Variables Checks
21/10/2024

Microsoft has announced TypeScript 5.7 in beta, with improvements including stronger checks for variables that have never been initialized before use, and path rewriting for relative paths.



Apache Lucene Improves Sparce Indexing
22/10/2024

Apache Lucene 10 has been released. The updated version adds a new IndexInput prefetch API, support for sparse indexing on doc values, and upgraded Snowball dictionaries resulting in improved tokeniza [ ... ]


More News

espbook

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 29 March 2021 )