Kafka Adds KRaft Migration
Written by Kay Ewbank   
Monday, 13 February 2023

Apache Kafka has been updated with improvements including the ability to to migrate Kafka clusters from ZK to KRaft mode with no downtime. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.

Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.

kafka

The main improvement to Kafka 3.4 is the ability to migrate Kafka clusters from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. The Kafka developers say this is an early access feature and they're looking for feedback.

The next improvement is an expansion of the metadata each group member passes to the group leader so that it's possible to check whether a partition is still owned and thus valid.

Other improvements add the option to disable JMX Reporter, and configure Kafka to disable JMX reporter for environments where it’s not being used. Producer IDs have also received attention, so you can now set a separate configuration for producer ID expiry, and can clean up producer IDs more efficiently to avoid excess memory usage.

Rack awareness for partitions has been added to this release, with new rack-aware partition assignment for Kafka Consumers, and for consumer rebalancing.

Kafka Connect has also received attention, with a new custom Kafka Admin interface for MirrorMaker2 that means users can run MirrorMaker2 with custom implementations for the Kafka resource manager. 

Kafka 3.4 is available now. 

kafka

More Information

Kafka Website

Related Articles

Kafka Adds KRaft-Based Authorizer  

Kafka 3.1 Adds OIDC Support

Kafka 3.0 Released With KRaft  

Kafka Replaces Zookeeper With Quorum

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Conference Times Ahead
29/03/2024

Following a well-established pattern both Google's and Microsoft's Developer Conferences will take place in May while Apple follows on in June. Here are the dates plus what to expect.



JetBrains Launches IDE Services
09/04/2024

JetBrains has launched a new product suite for enterprises. JetBrains IDE Services is designed for use by large organizations with the aim of boosting developer productivity at scale.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 13 February 2023 )