Kafka 3.0 Released With KRaft
Written by Kay Ewbank   
Friday, 24 September 2021

Apache has released Kafka 3.0 with new features, breaking API changes, and improvements to KRaft—Apache Kafka’s built-in consensus mechanism that will replace Apache ZooKeeper.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

kafka

It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

One major improvement to Apache 3.0 is the more mature version of KRaft, the internal Raft implementation that allows you to run Kafka brokers without Apache ZooKeeper. KRaft is not yet recommended for production, but the team says they have made many improvements to the KRaft metadata and APIs. A new major feature in 3.0 is the ability for KRaft Controllers and KRaft Brokers to generate, replicate, and load snapshots for the metadata topic partition that is used by the Kafka Cluster to store and replicate metadata information about the cluster like Broker configuration, topic partition assignment, and leadership,

Kafka Connect is another area to have been improved. Connect is a Kafka component that acts as a data hub for data integration between databases, key-value stores, search indexes, and file systems. The new version handles task restarts more smoothly. Until now, if you wanted to restart the whole connector, you had to make individual calls to restart the Connector instance and the Task instances. In 3.0, KIP-745 gives the ability to the users to restart either all or only the failed of a connector’s Connector and Task instances with a single call.

Kafka Streams has improvements in timestamp-based synchronization. The way Streams tasks fetch records has been enhanced, and this is supported by a new method in the Kafka Consumer API to deal locally with the consumer lag of a specific partition.

Another change of note is to MirrorMaker.This is a stand-alone tool for copying data between two Apache Kafka clusters, and it now has more flexible configuration options.

Kafka 3 is available now. 

 

kafka

More Information

Kafka Website

Related Articles

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

 

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


White House Urges Memory Safe Software
29/02/2024

The White House is urging developers to adopt memory safe programming languages, suggesting Rust would be a safer choice than C or C++. 



Ibis 8 Adds Streaming
05/03/2024

Ibis 8.0 has been released with stream processing backends. The new release includes Apache Flink as a streaming backend, and RisingWave, a streaming database backend. There's also a new batch backend [ ... ]


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info