Kafka 4.0 has been released, with major changes. This is the first version to operate entirely in KRaft mode by default. It also adds a new consumer group protocol designed to dramatically improve rebalance performance, and includes early access to Queues for Kafka, enabling Kafka to support traditional queue semantics directly. 

Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications. It was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

Apache Kafka 4.0 is a significant milestone, marking the first major release to operate entirely without Apache ZooKeeper. Instead, it runs in KRaft mode by default, simplifying deployment and management and enhancing scalability. 

Kafka 4.0 also introduces what the team says is a powerful new consumer group protocol designed to dramatically improve rebalance performance. They say this optimization significantly reduces downtime and latency, enhancing the reliability and responsiveness of consumer groups, especially in large-scale deployments.

The new version also includes early access to Queues for Kafka. This is not the traditional concept of a queue, but instead introduces the concept of a share group as a way of enabling cooperative consumption using Kafka topics. A share group is roughly equivalent to a "durable shared subscription" in existing systems. The Kafka Improvement Proposal about queues says this is queues done in a Kafka way, with no maximum queue depth and the ability to reset to a specific time for point-in-time recovery. The plan is that this will widen the uses for which Kafka can be a good messaging platform, particularly for those requiring point-to-point messaging patterns.

There are a number of other improvements, including ‎an extension for cluster operators to collect client metrics directly from brokers using a plugin. The previous extension only gathers metrics from Kafka clients (admin, consumer, and producer), and doesn't cover application-level metrics for embedded clients, such as Kafka Streams. This addition introduces a way for applications that use Kafka clients to include their own metrics alongside the existing client metrics.

Kafka 4.0 is available now. 

More Information

Kafka Website

Related Articles

Kafka 3.9 Adds Dynamic KRaft Quorums

Kafka Adds KRaft Migration

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info