Apache Ignite Adds Spark DataFrames Support
Written by Kay Ewbank   
Monday, 29 March 2021

Apache Ignite, a distributed database for high-performance computing with in-memory speed, has been updated with support for Spark DataFrames and machine learning.

Ignite can be used as a traditional SQL database via JDBC drivers, ODBC drivers, or its own native SQL APIs. By default, it runs purely in-memory, but clusters can be configured to run on a mix of disk and memory.


It supports 'co--located compute in Java, Scala, Kotlin, C#, and C++, meaning that you can create the equivalent of stored procedures in modern JVM languages, C# or C++ to develop and execute custom tasks across a distributed database. Ignite can operate in a strongly consistent mode that provides full support for distributed ACID transactions across multiple cluster nodes, caches, tables, and partitions.

Ignite supports continuous queries, meaning that rather than using a trigger to react to specific events, in Ignite you can write and run continuous queries written in languages such as Java or C# that process streams of changes on the database and application side.

Highlights of the latest version of the distributed database are general availability of its machine learning feature, and support for Spark DataFrames.

Ignite machine learning has TensorFlow integration as well as built-in algorithms and tools so it can be used to build scalable machine learning models without having to transfer data out of Ignite. You can train, deploy, evaluate, and update your ML and DL models continuously and at scale.

Perhaps even more strikingly, the latest version adds support for Spark DataFrames. The announcement emphasizes that this isn't a joke or misprint, saying that many early Ignite adopters have been hoping for this for years. A Spark DataFrame is a distributed collection of data organized into named columns, and means Spark can use the Catalyst query optimizer to produce more efficient query execution plans.

The developers say that Ignite expands DataFrame, simplifying development and improving data access times whenever Ignite is used as memory-centric storage for Spark. The support means Ignite users can share data and state across Spark jobs by writing and reading DataFrames to and from Ignite; and can improve the performance of SparkSQL queries by optimizing Spark query execution plans with the Ignite SQL engine which include​s advanced indexing. 


More Information

Apache Ignite Website

Related Articles

Spark 3 Improves Python and SQL Support

Apache Superset Reaches Top Level Project Status

Apache Daffodil Now Top Level Project

Facebook Apollo NoSQL Database  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Andrew Tanenbaum Gains ACM Award

Andrew Tanenbaum has been awarded the 2023 ACM System Software Award for MINIX the operating system he created for teaching purposes and which was an important influence on Linux.

ScyllaDB 6 Adds Node Distribution Feature

ScyllaDB 6.0 has been released with two major features that change the way it works: a dynamic way to distribute data across nodes that significantly improves scalability; and support for strongly con [ ... ]

More News

kotlin book



or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 29 March 2021 )