|Kafka Adds KRaft Migration|
|Written by Kay Ewbank|
|Monday, 13 February 2023|
Apache Kafka has been updated with improvements including the ability to to migrate Kafka clusters from ZK to KRaft mode with no downtime. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.
Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.
The main improvement to Kafka 3.4 is the ability to migrate Kafka clusters from ZooKeeper to KRaft mode. The migration copies the cluster metadata from ZooKeeper to the KRaft metadata log. During the migration, brokers are restarted in KRaft mode one at a time allowing the whole migration process to happen without cluster downtime. The Kafka developers say this is an early access feature and they're looking for feedback.
The next improvement is an expansion of the metadata each group member passes to the group leader so that it's possible to check whether a partition is still owned and thus valid.
Other improvements add the option to disable JMX Reporter, and configure Kafka to disable JMX reporter for environments where it’s not being used. Producer IDs have also received attention, so you can now set a separate configuration for producer ID expiry, and can clean up producer IDs more efficiently to avoid excess memory usage.
Rack awareness for partitions has been added to this release, with new rack-aware partition assignment for Kafka Consumers, and for consumer rebalancing.
Kafka Connect has also received attention, with a new custom Kafka Admin interface for MirrorMaker2 that means users can run MirrorMaker2 with custom implementations for the Kafka resource manager.
Kafka 3.4 is available now.
or email your comment to: email@example.com
|Last Updated ( Monday, 13 February 2023 )|