Apache Kafka, the distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications, has been updated with improvements including a KRaft-based authorizer and a proposal for marking KRaft mode as production ready in Apache Kafka 3.3.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

KRaft is Kafka Raft, the consensus protocol that has been added to remove Apache Kafka's dependency on ZooKeeper for metadata management. Until now, users making use of KRaft mode have still had to have an Apache ZooKeeper instance if they want to use AclAuthorizer. The new built-in Authorizer for KRaft mode doesn't depend on ZooKeeper, so avoids this.

This release also sees moves to using KRaft as production-ready. KRaft mode is popular as it addresses many urgent scalability and performance issues in Kafka, as well as eliminating the need to run an Apache ZooKeeper cluster alongside every Kafka cluster. KRaft entered preview status in Kafka 3.0, and this release opens the discussion on it becoming production-ready for new clusters in the upcoming Kafka 3.3 release. If this is agreed, ZooKeeper mode will be deprecated in Kafka 3.4 and removed entirely in Kafka 4.0.

Another improvement means the Kafka controller can send a hint to the partition leader to recover the partition in situations where the transaction state has been left inconsistent following an unclean election. Another change means that Kafka Streams can distribute its standby replicas over distinct “racks”.

Kafka 3.2 is available now. 

More Information

Kafka Website

Related Articles

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

Apache Kafka 2.7 Updates Broker

Kafka 2.5 Adds New Metrics And Improves Security

Kafka 2 Adds Support For ACLs

Kafka Graphs Framework Extends Kafka Streams

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info