|Kafka KRaft Now In Production|
|Written by Kay Ewbank|
|Thursday, 06 October 2022|
Apache Kafka has been updated with a new way to run Apache Kafka with self-managed metadata. Kafka is a distributed streaming platform that can be used for building real-time streaming data pipelines between systems or applications.
Kafka was originally developed at LinkedIn, from where it was taken on as an Apache project. It is a fast, scalable, durable, and fault-tolerant publish-subscribe messaging system that can be used in place of traditional message brokers.
The main improvement to Kafka is the marking of KRaft mode as production ready, though only for new clusters. This gives a way to run Apache Kafka with self-managed metadata, so eliminating the need to run an Apache Zookeeper cluster alongside every Kafka cluster. The new KRaft mode improves partition scalability and resiliency while simplifying deployments of Apache Kafka.
There are some features that are currently supported by Apache ZooKeeper mode that are not yet supported by KRaft mode. These include configuring SCRAM users via the administrative API, supporting JBOD configurations with multiple storage directories, and modifying certain dynamic configurations on the standalone KRaft controller. The plan is that these features will be added over the next few months.
Other additions include transactional EOS for source connectors. Exactly Once Semantics (EOS) is a framework that allows stream processing applications such as Kafka Streams to process data through Kafka without loss or duplication. The second improvement (which is also part of the work supporting EOS for source connectors) is support for atomic commit of source connector records and offsets, meaning that their offsets are committed if and only if all records up to that point have been ack'd by the producer.
The other new feature is the ability for the admin API to use batched offset fetch requests.
Kafka 3.3 is available now.
or email your comment to: firstname.lastname@example.org
|Last Updated ( Thursday, 13 October 2022 )|