Apache Ignite Adds Change Data Capture
Written by Kay Ewbank   
Thursday, 20 January 2022

Apache Ignite has been updated with improvements including Change Data Capture (CDC), an Index Query API, and several vulnerability fixes. Ignite is a distributed database for high-performance computing with in-memory speed.

Ignite can be used as a traditional SQL database via JDBC drivers, ODBC drivers, or its own native SQL APIs. By default, it runs purely in-memory, but clusters can be configured to run on a mix of disk and memory.

ignite

It supports co--located compute in Java, Scala, Kotlin, C#, and C++.

Change Data Capture (CDC) is a data processing pattern used to asynchronously receive entries that have been changed on the local node so that action can be taken using the changed entry.

The Ignite team says CDC is useful to handle streaming changes in a data warehouse, updating search indexes, and async interactions with external systems. It can also be used to calculate statistics for streaming queries. Ignite implements CDC with the ignite-cdc.sh application and Java API.

When CDC is enabled, the Ignite server node creates a hard link to each WAL archive segment in a special directory. The ignite-cdc.sh application can be runruns on a different JVM and processes newly archived WAL segments. When the segment is fully processed by ignite-cdc.sh, it is removed. WAL (write ahead log) archive segments hold the details of all data modifying operations (including deletes) that happen on a node.

Another improvement in this release is the provision of an Index Query API for existing indexes. Index queries work over distributed indexes and retrieve cache entries that match the specified query. The new API should help when SQL is not applicable because of the design of an application, or where IndexScan is preferable to ScanQuery for performance reasons.

Apache Ignite 2.12 is available now. ignite

More Information

Apache Ignite Website

Related Articles

Apache Ignite Adds Spark DataFrames Support

Spark 3 Improves Python and SQL Support

Apache Superset Reaches Top Level Project Status

Apache Daffodil Now Top Level Project

Facebook Apollo NoSQL Database 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Why OpenSSF's Baseline Security For Open Source Projects Is Important
21/04/2025

The Open Source Project Security Baseline, or OSPS Baseline for short, is a new initiative by OpenSSF in an attempt to bolster the security posture of open source software projects.



Google Adds Open-Source Development Kit To Vertex AI
15/04/2025

Google has added an Agent Development Kit (ADK) to Vertex AI, along with an agent engine and an Agent2Agent protocol that provides agents with a common, open language for collaboration.  The anno [ ... ]


More News

espbook

 

Comments




or email your comment to: comments@i-programmer.info