Apache Ignite Adds Change Data Capture
Written by Kay Ewbank   
Thursday, 20 January 2022

Apache Ignite has been updated with improvements including Change Data Capture (CDC), an Index Query API, and several vulnerability fixes. Ignite is a distributed database for high-performance computing with in-memory speed.

Ignite can be used as a traditional SQL database via JDBC drivers, ODBC drivers, or its own native SQL APIs. By default, it runs purely in-memory, but clusters can be configured to run on a mix of disk and memory.

ignite

It supports co--located compute in Java, Scala, Kotlin, C#, and C++.

Change Data Capture (CDC) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.

The Ignite team says CDC is useful to handle streaming changes in a data warehouse, updating search indexes, and async interactions with external systems. It can also be used to calculate statistics for streaming queries. Ignite implements CDC with the ignite-cdc.sh application and Java API.

When CDC is enabled, the Ignite server node creates a hard link to each WAL archive segment in a special directory. The ignite-cdc.sh application can be runruns on a different JVM and processes newly archived WAL segments. When the segment is fully processed by ignite-cdc.sh, it is removed. WAL (write ahead log) archive segments hold the details of all data modifying operations (including deletes) that happen on a node.

Another improvement in this release is the provision of an Index Query API for existing indexes. Index queries work over distributed indexes and retrieve cache entries that match the specified query. The new API should help when SQL is not applicable because of the design of an application, or where IndexScan is preferable to ScanQuery for performance reasons.

Apache Ignite 2.12 is available now. ignite

More Information

Apache Ignite Website

Related Articles

Apache Ignite Adds Spark DataFrames Support

Spark 3 Improves Python and SQL Support

Apache Superset Reaches Top Level Project Status

Apache Daffodil Now Top Level Project

Facebook Apollo NoSQL Database 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Kafka Adds KRaft-Based Authorizer
19/05/2022

Apache Kafka, the distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications, has been updated with improvements including a K [ ... ]



Let's Encrypt Awarded the Levchin Prize
22/04/2022

This year's recipient of the $10,000 prize for significant advances in the practice of cryptography and its use in real-world systems, is Let's Encrypt. Its Executive Director, Josh Aas  [ ... ]


More News

pythondata

 



 

Comments




or email your comment to: comments@i-programmer.info